Skip to content

Commit 9672d0b

Browse files
kylehounslowkylehounslow
committed
feat(compat): add vendor compatibility translation edge
Adds compat/ package and docker-compose.compat.yml overlay that accepts telemetry from Datadog, Jaeger (legacy wire protocol), and Splunk HEC agents and forwards OTLP to the base observability-stack collector. Includes customer-facing documentation under docs/starlight-docs and a Writing Tenets section in CONTRIBUTING.md. The overlay is opt-in and follows the existing INCLUDE_COMPOSE_* activation pattern. The base collector, its config, and all existing pipelines are unchanged. Activation: echo 'INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml' >> .env docker compose up -d Compat collector pipelines: - traces: [datadog, jaeger] - metrics: [datadog, statsd, splunk_hec] - logs: [datadog, splunk_hec] Modern OpenTelemetry SDK applications bypass the compat hop and send OTLP directly to the base collector on 4317/4318. Public documentation added under /docs/send-data/from-vendor/: - index.md (overview + architecture + decision table) - datadog.md, jaeger.md, splunk.md (per-vendor migration guides) - From Vendor Agents sidebar entry added via astro.config.mjs - Cross-link added from /docs/send-data/ overview Validated end-to-end with real vendor SDKs: - Datadog: dd-trace-py + FastAPI + patch_all() -> 42 spans in OpenSearch with service.name, service.version, deployment.environment.name preserved; instrumentationScope Datadog; parent/child chains intact - Splunk HEC: splunk_handler (Python logging) -> 5 log records in logs-otel-v1-* with com.splunk.source/sourcetype/index preserved - Jaeger OTLP: jaegertracing/example-hotrod demo -> 40 spans, 6-service topology, visible in APM and Discover Traces Starlight docs build validated (115 pages, all internal links resolve). SignalFx is not supported. The upstream signalfxreceiver is deprecated with explicit guidance to migrate to OTLP. Signed-off-by: Kyle Hounslow <kylhouns@amazon.com>
1 parent aa0fb14 commit 9672d0b

15 files changed

Lines changed: 794 additions & 0 deletions

File tree

‎CONTRIBUTING.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -398,6 +398,7 @@ AI coding assistants are welcome to contribute! When contributing as an AI agent
398398
- Note performance considerations
399399
- Reference relevant specifications
400400
- Keep comments up to date
401+
- Avoid narrative prose and internal team voice (e.g., "we deliberately...") — comments are for future maintainers, not decision history
401402

402403
### Examples
403404

@@ -407,6 +408,53 @@ AI coding assistants are welcome to contribute! When contributing as an AI agent
407408
- Follow language-specific conventions
408409
- Test examples before committing
409410

411+
### Writing Tenets (for agents)
412+
413+
Applies to all user-facing documentation: READMEs, public docs under `docs/`, migration guides, and PR descriptions.
414+
415+
**Framing**
416+
417+
- No us-vs-them. Write "Point your agents at observability-stack," not "at us." In open-source projects, the reader is part of the same community.
418+
- Don't leak internal conversation into docs. Design-doc voice ("this is the canonical case where...", "we deliberately omitted...") belongs in PR discussion or design documents, not user-facing artifacts.
419+
- Factual, not promotional. Avoid marketing phrases like "does ONE thing," "zero drift risk," or "honest limits."
420+
- Acknowledge nuance via asides (`:::note` in Starlight docs) or italic notes, not prose digressions.
421+
422+
**Maintenance hygiene**
423+
424+
- Don't pin version strings. Link to `main` of upstream repos (e.g., `opentelemetry-collector-contrib/tree/main/receiver/...`), not to a specific tag. Version pins go stale.
425+
- Don't duplicate source code in docs. Config YAML, pipeline definitions, and translation tables drift from the real source. Link to the source file instead.
426+
- Don't maintain per-vendor translation tables beyond 3–5 canonical well-known fields. Defer to upstream receiver READMEs and vendor documentation. Positioning this repo as a schema authority creates permanent maintenance burden.
427+
- Repo READMEs should link to public docs, not duplicate them. One source of truth per content type.
428+
429+
**Accuracy**
430+
431+
- Verify specific claims before writing them. Dates, version numbers, protocol behavior, UI terminology — check primary sources.
432+
- If a claim cannot be verified from primary sources, phrase it more vaguely. "Modern versions support X" beats "as of v1.42, X is supported" when the version claim is unverified.
433+
- Check existing conventions. Before using a UI name or terminology, grep the rest of the docs to see what other pages call it.
434+
- Run the documentation build (`npm run build` in `docs/starlight-docs`) before committing doc changes. Verify internal links are valid.
435+
436+
**Public-doc page structure**
437+
438+
Pages for users migrating TO observability-stack should cover, in order:
439+
440+
1. Action-oriented lead (one sentence — what the reader can do)
441+
2. Decision table when multiple paths exist ("Do I need this?" / "Which path applies?")
442+
3. Configuration — concrete environment variables, example config, code snippet per path
443+
4. Verify step — one-command check that it's working
444+
5. What lands in OpenSearch — concrete example of end state (field names, index patterns)
445+
6. Caveats — real observed gotchas surfaced from validation, not theoretical ones
446+
7. Not covered — honest scope boundaries
447+
8. References — upstream sources, vendor docs
448+
449+
**Repo READMEs** are for contributors, not migrators. Keep them short (20–40 lines for leaf READMEs; 100 max for overview). Link out to public docs for user-facing content. Include repo-local context only: config file paths, upstream receiver links, local dev workflow commands.
450+
451+
**Caveats from real validation are more trustworthy than theoretical ones.** When end-to-end testing reveals a gotcha (e.g., an attribute gets overwritten, a field doesn't translate), document it in the caveats section. Lead with what the user will see, not why it happens.
452+
453+
**Scope discipline**
454+
455+
- Prune aggressively when in doubt. Deletion is cheaper than maintenance.
456+
- Don't commit working files — audit tables, compatibility matrices, session notes, TODO lists, WIP drafts. If it's not useful to a future reader with no context, it's not a docs artifact.
457+
410458
## Community
411459

412460
### Getting Help

‎compat/README.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# Vendor Compatibility Overlay
2+
3+
User-facing migration guides: https://observability.opensearch.org/docs/send-data/from-vendor/
4+
5+
This overlay adds a dedicated OpenTelemetry Collector that accepts Datadog, Jaeger, and Splunk HEC wire protocols, translates them to OTLP, and forwards to the base collector. No application code changes are required — vendor agents are pointed at observability-stack by changing an endpoint URL.
6+
7+
## Do I need this overlay?
8+
9+
| If your apps emit... | Need this overlay? |
10+
|----------------------|--------------------|
11+
| OpenTelemetry OTLP (gRPC or HTTP) | No. Send directly to the base collector on 4317 or 4318. |
12+
| Datadog (dd-trace-*, DogStatsD) | Yes. |
13+
| Jaeger native wire protocol (`jaeger-client-*`) | Yes. |
14+
| Splunk HEC | Yes. |
15+
| Jaeger via modern OpenTelemetry SDK + OTLP | No. |
16+
17+
## Architecture
18+
19+
```
20+
vendor agents ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base, unchanged) ──▶ Data Prepper / Prometheus ──▶ OpenSearch
21+
(this overlay)
22+
23+
OTLP apps ─────────────────────────────────▶ (direct to base, no compat hop)
24+
```
25+
26+
The compat collector uses upstream [`opentelemetry-collector-contrib`](https://github.com/open-telemetry/opentelemetry-collector-contrib) receivers. All enrichment and downstream routing happens in the base pipeline — the compat config is purely ingest + forward.
27+
28+
## Activation
29+
30+
```bash
31+
echo "INCLUDE_COMPOSE_COMPAT=docker-compose.compat.yml" >> .env
32+
docker compose up -d
33+
```
34+
35+
Adds `otel-collector-compat` and the bundled Jaeger `hotrod` demo to the stack.
36+
37+
### Verify it's running
38+
39+
```bash
40+
docker compose ps otel-collector-compat
41+
curl -sI http://localhost:8126/info # HTTP 200 = Datadog receiver is live
42+
```
43+
44+
## Supported vendors
45+
46+
| Vendor | Receiver(s) | Default ports | Repo notes |
47+
|--------|-------------|---------------|------------|
48+
| Datadog | `datadogreceiver`, `statsdreceiver` | 8126/tcp, 8125/udp | [vendors/datadog/](vendors/datadog/) |
49+
| Jaeger (legacy wire protocol) | `jaegerreceiver` | 14250/tcp, 14268/tcp | [vendors/jaeger/](vendors/jaeger/) |
50+
| Splunk HEC | `splunkhecreceiver` | 8088/tcp | [vendors/splunk/](vendors/splunk/) |
51+
52+
User-facing migration guides live at https://observability.opensearch.org/docs/send-data/from-vendor/.
53+
54+
SignalFx is not supported. The upstream `signalfxreceiver` is deprecated with explicit guidance to migrate to OTLP.
55+
56+
## Deployment modes
57+
58+
Each vendor supports greenfield, side-by-side, and full-replacement modes. See the public migration guide for each vendor for specifics.
59+
60+
## Port customization
61+
62+
Ports are remappable via environment variables. Useful when a real vendor agent already occupies the default port on the host.
63+
64+
| Variable | Default | Receiver |
65+
|----------|---------|----------|
66+
| `COMPAT_DATADOG_APM_PORT` | 8126 | Datadog trace-agent |
67+
| `COMPAT_DATADOG_STATSD_PORT` | 8125 | DogStatsD |
68+
| `COMPAT_JAEGER_GRPC_PORT` | 14250 | Jaeger gRPC |
69+
| `COMPAT_JAEGER_THRIFT_HTTP_PORT` | 14268 | Jaeger Thrift HTTP |
70+
| `COMPAT_SPLUNK_HEC_PORT` | 8088 | Splunk HEC |
71+
| `COMPAT_COLLECTOR_MEMORY_LIMIT` | 256M | Compat collector memory limit |
72+
73+
## Attribute translation
74+
75+
Each receiver translates vendor-specific data to the OpenTelemetry data model. Translation behavior is defined by the upstream receivers. For schema details, consult:
76+
77+
- The upstream receiver READMEs under [`opentelemetry-collector-contrib/receiver/`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver)
78+
- The vendor's own instrumentation and tagging documentation
79+
80+
## Directory layout
81+
82+
```
83+
compat/
84+
├── README.md ← this file
85+
├── collector/
86+
│ ├── config.compat.yaml ← compat collector config
87+
│ └── README.md ← compat collector design notes
88+
└── vendors/
89+
├── datadog/README.md ← developer notes + link to migration guide
90+
├── jaeger/README.md
91+
└── splunk/README.md
92+
93+
docker-compose.compat.yml ← overlay service definitions
94+
```
95+
96+
## Adding a vendor
97+
98+
1. Create `vendors/<name>/README.md` with a link to the (forthcoming) public migration guide and developer notes (receiver used, config location, quick local test).
99+
2. Add the receiver stanza to `collector/config.compat.yaml` and wire it into the appropriate pipeline(s).
100+
3. Add port mappings to `docker-compose.compat.yml`.
101+
4. Add a page at `docs/starlight-docs/src/content/docs/send-data/from-vendor/<name>.md` with the user migration guide.
102+
5. Verify end-to-end: send vendor-format data → confirm it lands in OpenSearch.

‎compat/collector/README.md‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Compat Collector
2+
3+
Developer notes for `otel-collector-compat` and [`config.compat.yaml`](./config.compat.yaml).
4+
5+
For migration guides and usage, see the public docs: https://observability.opensearch.org/docs/send-data/from-vendor/
6+
7+
## Role
8+
9+
Accepts vendor wire protocols on their native ports, translates to the OpenTelemetry data model via upstream [`opentelemetry-collector-contrib`](https://github.com/open-telemetry/opentelemetry-collector-contrib) receivers, and forwards OTLP to the base collector. All enrichment, filtering, and downstream routing (Data Prepper, Prometheus) happens in the base collector config.
10+
11+
```
12+
vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
13+
[config.compat.yaml] [unchanged]
14+
```
15+
16+
The compat config contains only receivers, the `batch` processor, and an OTLP exporter pointed at the base collector. No transforms.
17+
18+
## Receivers
19+
20+
| Receiver | Upstream |
21+
|----------|----------|
22+
| `datadog` | [`datadogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver) |
23+
| `statsd` | [`statsdreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver) |
24+
| `jaeger` | [`jaegerreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver) |
25+
| `splunk_hec` | [`splunkhecreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver) |
26+
27+
Pipeline wiring lives in [`config.compat.yaml`](./config.compat.yaml) under `service.pipelines`.
28+
29+
## Local dev workflow
30+
31+
Edit `config.compat.yaml`, then:
32+
33+
```bash
34+
docker compose restart otel-collector-compat
35+
docker compose logs -f otel-collector-compat
36+
```
37+
38+
The `debug` exporter is wired into every pipeline. To see what's flowing through, bump its verbosity to `detailed`:
39+
40+
```yaml
41+
exporters:
42+
debug:
43+
verbosity: detailed
44+
```
45+
46+
## Resource limits
47+
48+
Default memory limit: 256MB (`COMPAT_COLLECTOR_MEMORY_LIMIT`). Idle usage is typically under 100MB.

‎compat/collector/config.compat.yaml‎

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
# config.compat.yaml
2+
#
3+
# Edge collector config. Translates vendor wire protocols to OTLP and forwards
4+
# to the base collector, which handles enrichment and downstream export.
5+
#
6+
# vendor apps ──▶ otel-collector-compat ──OTLP──▶ otel-collector (base)
7+
# (this config) (unchanged)
8+
9+
receivers:
10+
# Datadog trace-agent protocol. Also accepts metrics and logs endpoints;
11+
# all three are wired into the service pipelines below.
12+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver
13+
datadog:
14+
endpoint: 0.0.0.0:8126
15+
read_timeout: 60s
16+
trace_id_cache_size: 100
17+
18+
# StatsD / DogStatsD over UDP.
19+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver
20+
statsd:
21+
endpoint: 0.0.0.0:8125
22+
aggregation_interval: 60s
23+
enable_metric_type: true
24+
is_monotonic_counter: false
25+
26+
# Jaeger native wire protocol (Thrift HTTP + gRPC). For legacy
27+
# jaeger-client-* applications. Modern Jaeger apps emit OTLP and send
28+
# directly to the base collector on 4317/4318 without this hop.
29+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver
30+
jaeger:
31+
protocols:
32+
grpc:
33+
endpoint: 0.0.0.0:14250
34+
thrift_http:
35+
endpoint: 0.0.0.0:14268
36+
37+
# Splunk HTTP Event Collector (logs and metrics).
38+
# See: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver
39+
splunk_hec:
40+
endpoint: 0.0.0.0:8088
41+
42+
processors:
43+
# Batch spans/events to reduce RPC volume against the base collector.
44+
batch:
45+
timeout: 10s
46+
send_batch_size: 1024
47+
48+
exporters:
49+
# Forward to the base collector. Downstream routing (Data Prepper,
50+
# Prometheus, etc.) is configured there.
51+
otlp_grpc:
52+
endpoint: otel-collector:4317
53+
tls:
54+
insecure: true
55+
56+
debug:
57+
verbosity: basic
58+
59+
service:
60+
pipelines:
61+
traces:
62+
receivers: [datadog, jaeger]
63+
processors: [batch]
64+
exporters: [otlp_grpc, debug]
65+
66+
metrics:
67+
receivers: [datadog, statsd, splunk_hec]
68+
processors: [batch]
69+
exporters: [otlp_grpc, debug]
70+
71+
logs:
72+
receivers: [datadog, splunk_hec]
73+
processors: [batch]
74+
exporters: [otlp_grpc, debug]
75+
76+
telemetry:
77+
logs:
78+
level: info

‎compat/vendors/datadog/README.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Datadog
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/datadog/
4+
5+
## Receivers used
6+
7+
- [`datadogreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/datadogreceiver) — traces, metrics, logs (8126/tcp)
8+
- [`statsdreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/statsdreceiver) — DogStatsD (8125/udp)
9+
10+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml) — `datadog:` and `statsd:` receiver blocks.
11+
12+
## Quick local test
13+
14+
```bash
15+
# DogStatsD metric
16+
echo "test.metric:1|c|#env:dev" | nc -u -w1 localhost 8125
17+
18+
# HEC-style trace payload (msgpack) — see upstream receiver README for format details
19+
```
20+
21+
Traces are best exercised by pointing a `dd-trace-*` SDK application at `localhost:8126`.

‎compat/vendors/jaeger/README.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
# Jaeger
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/jaeger/
4+
5+
Jaeger is an open-source CNCF project rather than a proprietary vendor. This directory exists because users migrating from Jaeger deployments typically look for it alongside other vendor integrations.
6+
7+
## Receiver used
8+
9+
- [`jaegerreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/jaegerreceiver) — Jaeger native wire protocol (Thrift HTTP on 14268/tcp, gRPC on 14250/tcp)
10+
11+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml) — `jaeger:` receiver block.
12+
13+
Modern Jaeger apps emit OTLP natively and bypass this overlay. The bundled `hotrod` demo (port 8080) is configured this way and exercises the base collector directly.
14+
15+
## Quick local test
16+
17+
```bash
18+
# Trigger the bundled hotrod demo (OTLP path)
19+
curl http://localhost:8080/dispatch?customer=123
20+
21+
# View traces at http://localhost:5601 → Trace Analytics
22+
```
23+
24+
The legacy Thrift HTTP path is harder to exercise without a `jaeger-client-*` app. See the upstream receiver README for wire format details.

‎compat/vendors/splunk/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
# Splunk HEC
2+
3+
Migration guide and user-facing documentation: https://observability.opensearch.org/docs/send-data/from-vendor/splunk/
4+
5+
## Receiver used
6+
7+
- [`splunkhecreceiver`](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/splunkhecreceiver) — Splunk HTTP Event Collector (8088/tcp)
8+
9+
Config: [`../../collector/config.compat.yaml`](../../collector/config.compat.yaml) — `splunk_hec:` receiver block.
10+
11+
Wired into the logs and metrics pipelines. Traces are not wired in the default compat config.
12+
13+
## Quick local test
14+
15+
```bash
16+
curl -X POST http://localhost:8088/services/collector \
17+
-H "Authorization: Splunk any-token" \
18+
-H "Content-Type: application/json" \
19+
-d '{"event":"hello","sourcetype":"manual","source":"curl"}'
20+
# {"text": "Success", "code": 0}
21+
```
22+
23+
Events land in OpenSearch under `logs-otel-v1-*`. View via Discover Logs in OpenSearch Dashboards.

0 commit comments

Comments
 (0)