docs(otel): add contrib collector to self-managed Elastic guide#7032
docs(otel): add contrib collector to self-managed Elastic guide#7032alexandra5000 wants to merge 7 commits into
Conversation
Elastic Docs AI PR menuCheck the box to run an AI review for this pull request.
Powered by GitHub Agentic Workflows and docs-actions. For more information, reach out to the docs team. |
✅ Elastic Docs Style Checker (Vale)No issues found on modified lines! The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale. |
| export ELASTIC_API_KEY=your-encoded-api-key | ||
| ``` | ||
|
|
||
| Then create `gateway.yml`: |
There was a problem hiding this comment.
I based the gateway.yml on: elastic-agent/internal/edot/samples/linux/gateway.yml
Removes the "Deploy on k8s" section from the bare-metal guide and replaces it with a Next steps link to the upcoming kubernetes.md page. Keeps the bare-metal guide self-contained so it can merge independently. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… a mode Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
| batch: | ||
| send_batch_size: 1000 | ||
| timeout: 1s | ||
| send_batch_max_size: 1500 | ||
| batch/metrics: | ||
| send_batch_max_size: 0 | ||
| timeout: 1s |
There was a problem hiding this comment.
Can we get rid of the Batch processor in favor of batching in Elasticsearch exporter?
The Batch processor in current form is buggy. See:
- [processor/batch] Batch processor drops data that failed to be sent open-telemetry/opentelemetry-collector#12443
- [batchprocessor] Modernize or remove the Batch processor open-telemetry/opentelemetry-collector#13582
- Resolution: Batching processor and/or connector still needed open-telemetry/opentelemetry-collector#15047
There was a problem hiding this comment.
Thanks @andrzej-stencel, that's a good reason to get rid of it. Could you drop the replacement config here? I want to make sure we recommend the right ES exporter batching settings, and I'm not confident enough in the exact parameter names to write it without your confirmation.
Another thing to take into consideration is that with this change we'll diverge from the elastic-agent/internal/edot/samples/linux/gateway.yml sample - but I'm guessing that will be updated eventually? In such a case I think it's ok to diverge, even if it's not great to have docs say something different from what you get from the actual EDOT repo...
| - id: edot-collector | ||
| --- | ||
|
|
||
| # Send data from a contrib OpenTelemetry Collector [upstream-collector-self-managed] |
There was a problem hiding this comment.
Why Contrib specifically? OpenTelemetry only has several distros: https://opentelemetry.io/docs/collector/distributions/
I would frame it just OpenTelemetry Collector (i.e. contrib) to make it more generic and also cover for the cases where users build their own distros using ocb. We can then say sth like "for the shake of this example we will be using contrib for simplicity".
3 participants
Summary
Closes #6972
Adds a how-to guide for forwarding telemetry from an upstream contrib OpenTelemetry Collector to a self-managed Elastic Stack using an EDOT Collector gateway:
contrib otelcol-contrib → EDOT Collector (gateway mode) → Elasticsearch
gateway.ymlsample in elastic-agentGenerative AI disclosure
Did you use a generative AI (GenAI) tool to assist in creating this contribution?
If you answered "Yes" to the previous question, please specify the tool(s) and model(s) used (e.g., Google Gemini, OpenAI ChatGPT-4, etc.).
Tool(s) and model(s) used: Claude Opus 4.8