Skip to content

Commit f06f47f

Browse files
committed
feat(compat): add vendor compatibility translation edge
Adds compat/ package and docker-compose.compat.yml overlay that accepts telemetry from Datadog, Jaeger (legacy wire protocol), and Splunk HEC agents and forwards OTLP to the base observability-stack collector. Includes customer-facing documentation under docs/starlight-docs. Architecture: dedicated otel-collector-compat edge collector runs the upstream collector-contrib receivers for each vendor, batches, and forwards OTLP to the base collector — which handles all existing processor/exporter logic unchanged. No drift risk with the base config. Modern OTel-SDK apps (including Jaeger v1.42+) bypass the compat hop and send OTLP directly to the base collector. Activation uses the repo's existing INCLUDE_COMPOSE_* convention: echo 'INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml' >> .env docker compose up -d Pipelines wired into compat collector: - traces: [datadog, jaeger] - metrics: [datadog, statsd, splunk_hec] - logs: [datadog, splunk_hec] Public documentation added under /docs/send-data/from-vendor/: - index.md (overview + architecture + quickstart) - datadog.md, jaeger.md, splunk.md (per-vendor migration guides) - 'From Vendor Agents' sidebar entry added via astro.config.mjs - Link added from /docs/send-data/ overview Scope and framing: - This is protocol-compatible ingest, not a 1:1 vendor platform replacement. - We defer to upstream collector-contrib READMEs and vendor documentation for attribute schemas and translation behavior. The vendor pages here document only 3-5 canonical attributes per vendor and link out to the authoritative sources. - Component receivers have varying stability tiers upstream (alpha/beta/ development). Vendor pages document the specifics. SignalFx deliberately omitted — upstream signalfxreceiver was deprecated 2026-02-13 with explicit guidance to migrate to OTLP. Bundles Jaeger's hotrod demo on port 8080 as a built-in OTLP trace generator (Jaeger's canonical demo, pointed at the base collector). Validated: starlight-docs astro build passes with all internal links valid (115 pages built successfully). Signed-off-by: Kyle Hounslow <kylhouns@amazon.com>
1 parent aa0fb14 commit f06f47f

15 files changed

Lines changed: 794 additions & 0 deletions

File tree

CONTRIBUTING.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -398,6 +398,7 @@ AI coding assistants are welcome to contribute! When contributing as an AI agent
398398
- Note performance considerations
399399
- Reference relevant specifications
400400
- Keep comments up to date
401+
- Avoid narrative prose and internal team voice (e.g., "we deliberately...") — comments are for future maintainers, not decision history
401402

402403
### Examples
403404

@@ -407,6 +408,53 @@ AI coding assistants are welcome to contribute! When contributing as an AI agent
407408
- Follow language-specific conventions
408409
- Test examples before committing
409410

411+
### Writing Tenets (for agents)
412+
413+
Applies to all user-facing documentation: READMEs, public docs under `docs/`, migration guides, and PR descriptions.
414+
415+
**Framing**
416+
417+
- No us-vs-them. Write "Point your agents at observability-stack," not "at us." In open-source projects, the reader is part of the same community.
418+
- Don't leak internal conversation into docs. Design-doc voice ("this is the canonical case where...", "we deliberately omitted...") belongs in PR discussion or design documents, not user-facing artifacts.
419+
- Factual, not promotional. Avoid marketing phrases like "does ONE thing," "zero drift risk," or "honest limits."
420+
- Acknowledge nuance via asides (`:::note` in Starlight docs) or italic notes, not prose digressions.
421+
422+
**Maintenance hygiene**
423+
424+
- Don't pin version strings. Link to `main` of upstream repos (e.g., `opentelemetry-collector-contrib/tree/main/receiver/...`), not to a specific tag. Version pins go stale.
425+
- Don't duplicate source code in docs. Config YAML, pipeline definitions, and translation tables drift from the real source. Link to the source file instead.
426+
- Don't maintain per-vendor translation tables beyond 3–5 canonical well-known fields. Defer to upstream receiver READMEs and vendor documentation. Positioning this repo as a schema authority creates permanent maintenance burden.
427+
- Repo READMEs should link to public docs, not duplicate them. One source of truth per content type.
428+
429+
**Accuracy**
430+
431+
- Verify specific claims before writing them. Dates, version numbers, protocol behavior, UI terminology — check primary sources.
432+
- If a claim cannot be verified from primary sources, phrase it more vaguely. "Modern versions support X" beats "as of v1.42, X is supported" when the version claim is unverified.
433+
- Check existing conventions. Before using a UI name or terminology, grep the rest of the docs to see what other pages call it.
434+
- Run the documentation build (`npm run build` in `docs/starlight-docs`) before committing doc changes. Verify internal links are valid.
435+
436+
**Public-doc page structure**
437+
438+
Pages for users migrating TO observability-stack should cover, in order:
439+
440+
1. Action-oriented lead (one sentence — what the reader can do)
441+
2. Decision table when multiple paths exist ("Do I need this?" / "Which path applies?")
442+
3. Configuration — concrete environment variables, example config, code snippet per path
443+
4. Verify step — one-command check that it's working
444+
5. What lands in OpenSearch — concrete example of end state (field names, index patterns)
445+
6. Caveats — real observed gotchas surfaced from validation, not theoretical ones
446+
7. Not covered — honest scope boundaries
447+
8. References — upstream sources, vendor docs
448+
449+
**Repo READMEs** are for contributors, not migrators. Keep them short (20–40 lines for leaf READMEs; 100 max for overview). Link out to public docs for user-facing content. Include repo-local context only: config file paths, upstream receiver links, local dev workflow commands.
450+
451+
**Caveats from real validation are more trustworthy than theoretical ones.** When end-to-end testing reveals a gotcha (e.g., an attribute gets overwritten, a field doesn't translate), document it in the caveats section. Lead with what the user will see, not why it happens.
452+
453+
**Scope discipline**
454+
455+
- Prune aggressively when in doubt. Deletion is cheaper than maintenance.
456+
- Don't commit working files — audit tables, compatibility matrices, session notes, TODO lists, WIP drafts. If it's not useful to a future reader with no context, it's not a docs artifact.
457+
410458
## Community
411459

412460
### Getting Help

compat/README.md

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Vendor Compatibility Overlay
2+
3+
User-facing migration guides: https://observability.opensearch.org/docs/send-data/from-vendor/
4+
5+
This overlay adds a dedicated OpenTelemetry Collector that accepts Datadog, Jaeger, and Splunk HEC wire protocols, translates them to OTLP, and forwards to the base collector. No application code changes are required — vendor agents are pointed at observability-stack by changing an endpoint URL.
6+
7+
## Do I need this overlay?
8+
9+
| If your apps emit... | Need this overlay? |
10+
|----------------------|--------------------|
11+
| OpenTelemetry OTLP (gRPC or HTTP) | No. Send directly to the base collector on 4317 or 4318. |
12+
| Datadog (dd-trace-*, DogStatsD) | Yes. |
13+
| Jaeger native wire protocol (`jaeger-client-*`) | Yes. |
14+
| Splunk HEC | Yes. |
15+
| Jaeger via modern OpenTelemetry SDK + OTLP | No. |
16+
17+
## Architecture
18+
19+
```
20+
vendor agents ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base, unchanged) ──▶ Data Prepper / Prometheus ──▶ OpenSearch
21+
(this overlay)
22+
23+
OTLP apps ─────────────────────────────────▶ (direct to base, no compat hop)
24+
```
25+
26+
The compat collector uses upstream [`opentelemetry-collector-contrib`](https://github.com/open-telemetry/opentelemetry-collector-contrib) receivers. All enrichment and downstream routing happens in the base pipeline — the compat config is purely ingest + forward.
27+
28+
## Activation
29+
30+
```bash
31+
echo "INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml" >> .env
32+
docker compose up -d
33+
```
34+
35+
Adds `otel-collector-compat` and the bundled Jaeger `hotrod` demo to the stack.
36+
37+
### Verify it's running
38+
39+
```bash
40+
docker compose ps otel-collector-compat
41+
curl -sI http://localhost:8126/info # HTTP 200 = Datadog receiver is live
42+
```
43+
44+
## Supported vendors
45+
46+
| Vendor | Receiver(s) | Default ports | Repo notes |
47+
|--------|-------------|---------------|------------|
48+
| Datadog | `datadogreceiver`, `statsdreceiver` | 8126/tcp, 8125/udp | [vendors/datadog/](vendors/datadog/) |
49+
| Jaeger (legacy wire protocol) | `jaegerreceiver` | 14250/tcp, 14268/tcp | [vendors/jaeger/](vendors/jaeger/) |
50+
| Splunk HEC | `splunkhecreceiver` | 8088/tcp | [vendors/splunk/](vendors/splunk/) |
51+
52+
User-facing migration guides live at https://observability.opensearch.org/docs/send-data/from-vendor/.
53+
54+
SignalFx is not supported. The upstream `signalfxreceiver` is deprecated with explicit guidance to migrate to OTLP.
55+
56+
## Deployment modes
57+
58+
Each vendor supports greenfield, side-by-side, and full-replacement modes. See the public migration guide for each vendor for specifics.
59+
60+
## Port customization
61+
62+
Ports are remappable via environment variables. Useful when a real vendor agent already occupies the default port on the host.
63+
64+
| Variable | Default | Receiver |
65+
|----------|---------|----------|
66+
| `COMPAT_DATADOG_APM_PORT` | 8126 | Datadog trace-agent |
67+
| `COMPAT_DATADOG_STATSD_PORT` | 8125 | DogStatsD |
68+
| `COMPAT_JAEGER_GRPC_PORT` | 14250 | Jaeger gRPC |
69+
| `COMPAT_JAEGER_THRIFT_HTTP_PORT` | 14268 | Jaeger Thrift HTTP |
70+
| `COMPAT_SPLUNK_HEC_PORT` | 8088 | Splunk HEC |
71+
| `COMPAT_COLLECTOR_MEMORY_LIMIT` | 256M | Compat collector memory limit |
72+
73+
## Attribute translation
74+
75+
Each receiver translates vendor-specific data to the OpenTelemetry data model. Translation behavior is defined by the upstream receivers. For schema details, consult:
76+
77+
- The upstream receiver READMEs under [`opentelemetry-collector-contrib/receiver/`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver)
78+
- The vendor's own instrumentation and tagging documentation
79+
80+
## Directory layout
81+
82+
```
83+
compat/
84+
├── README.md ← this file
85+
├── collector/
86+
│ ├── config.compat.yaml ← compat collector config
87+
│ └── README.md ← compat collector design notes
88+
└── vendors/
89+
├── datadog/README.md ← developer notes + link to migration guide
90+
├── jaeger/README.md
91+
└── splunk/README.md
92+
93+
docker-compose.compat.yml ← overlay service definitions
94+
```
95+
96+
## Adding a vendor
97+
98+
1. Create `vendors/<name>/README.md` with a link to the (forthcoming) public migration guide and developer notes (receiver used, config location, quick local test).
99+
2. Add the receiver stanza to `collector/config.compat.yaml` and wire it into the appropriate pipeline(s).
100+
3. Add port mappings to `docker-compose.compat.yml`.
101+
4. Add a page at `docs/starlight-docs/src/content/docs/send-data/from-vendor/<name>.md` with the user migration guide.
102+
5. Verify end-to-end: send vendor-format data → confirm it lands in OpenSearch.

compat/collector/README.md

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Compat Collector
2+
3+
Developer notes for `otel-collector-compat` and [`config.compat.yaml`](./config.compat.yaml).
4+
5+
For migration guides and usage, see the public docs: https://observability.opensearch.org/docs/send-data/from-vendor/
6+
7+
## Role
8+
9+
Accepts vendor wire protocols on their native ports, translates to the OpenTelemetry data model via upstream [`opentelemetry-collector-contrib`](https://github.com/open-telemetry/opentelemetry-collector-contrib) receivers, and forwards OTLP to the base collector. All enrichment, filtering, and downstream routing (Data Prepper, Prometheus) happens in the base collector config.
10+
11+
```
12+
vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
13+
[config.compat.yaml] [unchanged]
14+
```
15+
16+
The compat config contains only receivers, the `batch` processor, and an OTLP exporter pointed at the base collector. No transforms.
17+
18+
## Receivers
19+
20+
| Receiver | Upstream |
21+
|----------|----------|
22+
| `datadog` | [`datadogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver) |
23+
| `statsd` | [`statsdreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver) |
24+
| `jaeger` | [`jaegerreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver) |
25+
| `splunk_hec` | [`splunkhecreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver) |
26+
27+
Pipeline wiring lives in [`config.compat.yaml`](./config.compat.yaml) under `service.pipelines`.
28+
29+
## Local dev workflow
30+
31+
Edit `config.compat.yaml`, then:
32+
33+
```bash
34+
docker compose restart otel-collector-compat
35+
docker compose logs -f otel-collector-compat
36+
```
37+
38+
The `debug` exporter is wired into every pipeline. To see what's flowing through, bump its verbosity to `detailed`:
39+
40+
```yaml
41+
exporters:
42+
debug:
43+
verbosity: detailed
44+
```
45+
46+
## Resource limits
47+
48+
Default memory limit: 256MB (`COMPAT_COLLECTOR_MEMORY_LIMIT`). Idle usage is typically under 100MB.
Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# config.compat.yaml
2+
#
3+
# Edge collector config. Translates vendor wire protocols to OTLP and forwards
4+
# to the base collector, which handles enrichment and downstream export.
5+
#
6+
# vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
7+
# (this config) (unchanged)
8+
9+
receivers:
10+
# Datadog trace-agent protocol. Also accepts metrics and logs endpoints;
11+
# all three are wired into the service pipelines below.
12+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver
13+
datadog:
14+
endpoint: 0.0.0.0:8126
15+
read_timeout: 60s
16+
trace_id_cache_size: 100
17+
18+
# StatsD / DogStatsD over UDP.
19+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver
20+
statsd:
21+
endpoint: 0.0.0.0:8125
22+
aggregation_interval: 60s
23+
enable_metric_type: true
24+
is_monotonic_counter: false
25+
26+
# Jaeger native wire protocol (Thrift HTTP + gRPC). For legacy
27+
# jaeger-client-* applications. Modern Jaeger apps emit OTLP and send
28+
# directly to the base collector on 4317/4318 without this hop.
29+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver
30+
jaeger:
31+
protocols:
32+
grpc:
33+
endpoint: 0.0.0.0:14250
34+
thrift_http:
35+
endpoint: 0.0.0.0:14268
36+
37+
# Splunk HTTP Event Collector (logs and metrics).
38+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver
39+
splunk_hec:
40+
endpoint: 0.0.0.0:8088
41+
42+
processors:
43+
# Batch spans/events to reduce RPC volume against the base collector.
44+
batch:
45+
timeout: 10s
46+
send_batch_size: 1024
47+
48+
exporters:
49+
# Forward to the base collector. Downstream routing (Data Prepper,
50+
# Prometheus, etc.) is configured there.
51+
otlp_grpc:
52+
endpoint: otel-collector:4317
53+
tls:
54+
insecure: true
55+
56+
debug:
57+
verbosity: basic
58+
59+
service:
60+
pipelines:
61+
traces:
62+
receivers: [datadog, jaeger]
63+
processors: [batch]
64+
exporters: [otlp_grpc, debug]
65+
66+
metrics:
67+
receivers: [datadog, statsd, splunk_hec]
68+
processors: [batch]
69+
exporters: [otlp_grpc, debug]
70+
71+
logs:
72+
receivers: [datadog, splunk_hec]
73+
processors: [batch]
74+
exporters: [otlp_grpc, debug]
75+
76+
telemetry:
77+
logs:
78+
level: info

compat/vendors/datadog/README.md

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Datadog
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/datadog/
4+
5+
## Receivers used
6+
7+
- [`datadogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver) — traces, metrics, logs (8126/tcp)
8+
- [`statsdreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver) — DogStatsD (8125/udp)
9+
10+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml)`datadog:` and `statsd:` receiver blocks.
11+
12+
## Quick local test
13+
14+
```bash
15+
# DogStatsD metric
16+
echo "test.metric:1|c|#env:dev" | nc -u -w1 localhost 8125
17+
18+
# HEC-style trace payload (msgpack) — see upstream receiver README for format details
19+
```
20+
21+
Traces are best exercised by pointing a `dd-trace-*` SDK application at `localhost:8126`.

compat/vendors/jaeger/README.md

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Jaeger
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/jaeger/
4+
5+
Jaeger is an open-source CNCF project rather than a proprietary vendor. This directory exists because users migrating from Jaeger deployments typically look for it alongside other vendor integrations.
6+
7+
## Receiver used
8+
9+
- [`jaegerreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver) — Jaeger native wire protocol (Thrift HTTP on 14268/tcp, gRPC on 14250/tcp)
10+
11+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml)`jaeger:` receiver block.
12+
13+
Modern Jaeger apps emit OTLP natively and bypass this overlay. The bundled `hotrod` demo (port 8080) is configured this way and exercises the base collector directly.
14+
15+
## Quick local test
16+
17+
```bash
18+
# Trigger the bundled hotrod demo (OTLP path)
19+
curl http://localhost:8080/dispatch?customer=123
20+
21+
# View traces at http://localhost:5601 → Trace Analytics
22+
```
23+
24+
The legacy Thrift HTTP path is harder to exercise without a `jaeger-client-*` app. See the upstream receiver README for wire format details.

compat/vendors/splunk/README.md

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Splunk HEC
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/splunk/
4+
5+
## Receiver used
6+
7+
- [`splunkhecreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver) — Splunk HTTP Event Collector (8088/tcp)
8+
9+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml)`splunk_hec:` receiver block.
10+
11+
Wired into the logs and metrics pipelines. Traces are not wired in the default compat config.
12+
13+
## Quick local test
14+
15+
```bash
16+
curl -X POST http://localhost:8088/services/collector \
17+
-H "Authorization: Splunk any-token" \
18+
-H "Content-Type: application/json" \
19+
-d '{"event":"hello","sourcetype":"manual","source":"curl"}'
20+
# {"text": "Success", "code": 0}
21+
```
22+
23+
Events land in OpenSearch under `logs-otel-v1-*`. View via Discover Logs in OpenSearch Dashboards.

0 commit comments

Comments
 (0)