Skip to content

Commit 2ff8326

Browse files
ps48ps48
committed
update and restructure APM docs
Signed-off-by: ps48 <pshenoy36@gmail.com>
1 parent 00175d9 commit 2ff8326

7 files changed

Lines changed: 284 additions & 119 deletions

File tree

‎docs/starlight-docs/public/images/apm/apm-settings-modal.png‎

321 KB
Loading

‎docs/starlight-docs/public/images/apm/application-map-node-details.png‎

758 KB
Loading

‎docs/starlight-docs/src/content/docs/apm/configuring-apm.md‎

Lines changed: 77 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,77 @@
1+
---
2+
title: Configuring APM in OpenSearch Dashboards
3+
description: Create datasets, index patterns, and connect data sources to enable APM features
4+
sidebar:
5+
label: Setting Up APM
6+
order: 15
7+
---
8+
9+
After ingesting telemetry data using the OpenTelemetry Collector and Data Prepper, you need to configure APM in OpenSearch Dashboards. You'll create datasets, configure index patterns, attach a Prometheus data source, and configure APM settings in your workspace.
10+
11+
## Step 1: Create datasets for logs and traces
12+
13+
Create datasets in your Observability workspace to define how APM accesses your trace and log data:
14+
15+
1. Navigate to your Observability workspace.
16+
2. Select **Datasets** and then select **Create dataset**.
17+
3. Create a **traces dataset** pointing to your trace data index (for example, `otel-v1-apm-span-*`).
18+
4. Create a **logs dataset** pointing to your application logs index (for example, `logs-otel-v1-*`).
19+
20+
For detailed instructions on creating and configuring datasets, see [Datasets](/docs/investigate/datasets/).
21+
22+
## Step 2: Create a correlation between trace and log datasets
23+
24+
Link the trace and log datasets so that APM can display related logs when you investigate traces:
25+
26+
1. Navigate to **Datasets** > **Correlations**.
27+
2. Select the trace dataset you created in [Step 1](#step-1-create-datasets-for-logs-and-traces).
28+
3. Add the logs dataset as a correlated dataset.
29+
4. Save the correlation.
30+
31+
This enables in-context log correlations throughout the APM interface, including the **Services** and **Application Map** pages. For detailed instructions, see [Correlations](/docs/investigate/correlations/).
32+
33+
## Step 3: Create an index pattern for the service map
34+
35+
APM requires an index pattern for the service map data generated by the `otel_apm_service_map` processor in Data Prepper. To create an index pattern, follow these steps:
36+
37+
1. Navigate to **Dashboards Management** > **Index patterns**.
38+
2. Select **Create index pattern**.
39+
3. Enter `otel-v2-apm-service-map*` as the index pattern name.
40+
4. Select the appropriate time field and save.
41+
42+
This index pattern is used by APM to display service topology data and dependency information.
43+
44+
## Step 4: Attach a Prometheus data source to your workspace
45+
46+
APM uses Prometheus to store and query Rate, Errors, Duration (RED) metrics. To add a Prometheus data source to your workspace, follow these steps:
47+
48+
1. Navigate to **Dashboards Management** > **Data sources**.
49+
2. Select **Create data source** and choose **Prometheus**.
50+
3. Enter the connection details for your Prometheus instance (for example, `http://prometheus:9090`).
51+
4. Save the data source and attach it to your Observability workspace.
52+
53+
For detailed instructions on configuring a Prometheus data source, see [Discover Metrics](/docs/investigate/discover-metrics/).
54+
55+
## Step 5: Configure APM settings
56+
57+
To configure APM settings, follow these steps:
58+
59+
1. Navigate to **APM** > **Services** or **APM** > **Application Map**.
60+
2. Select the **APM Settings** button in the upper-right corner. The **Application monitoring settings** dialog appears, as shown in the following image.
61+
62+
![APM settings modal](/docs/images/apm/apm-settings-modal.png)
63+
64+
3. In the **Application monitoring settings** dialog, configure the following settings:
65+
66+
- **Traces**: Select the trace dataset you created in [Step 1](#step-1-create-datasets-for-logs-and-traces) (for example, `Trace Dataset - Local Cluster`). The number of correlated logs is displayed below the selection.
67+
- **Services**: Select the service map index pattern you created in [Step 3](#step-3-create-an-index-pattern-for-the-service-map) (for example, `otel-v2-apm-service-map*`).
68+
- **RED Metrics**: Select the Prometheus data source you configured in [Step 4](#step-4-attach-a-prometheus-data-source-to-your-workspace) (for example, `ObservabilityStack_Prometheus`).
69+
70+
4. Select **Update** to save your configuration.
71+
72+
After completing this setup, APM will display service topology, RED metrics, and in-context correlations across the Services and Application Map pages.
73+
74+
## Next steps
75+
76+
- [Services](/docs/apm/services/): View service performance metrics and dependencies.
77+
- [Application Map](/docs/apm/service-map/): Explore the service topology visualization.

‎docs/starlight-docs/src/content/docs/apm/configuring-telemetry-ingestion.md‎

Lines changed: 177 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,177 @@
1+
---
2+
title: Configuring Telemetry Ingestion
3+
description: Set up the OpenTelemetry Collector and Data Prepper to ingest traces, logs, and metrics for APM
4+
sidebar:
5+
label: Telemetry Ingestion
6+
order: 10
7+
---
8+
9+
To use APM, you need to ingest application traces and logs into OpenSearch using the OpenTelemetry Collector and Data Prepper pipeline. For an overview of the complete APM architecture, see the [APM overview](/docs/apm/).
10+
11+
This page covers configuring the OpenTelemetry Collector and Data Prepper to process and route telemetry data to OpenSearch and Prometheus.
12+
13+
## Configuring the OpenTelemetry Collector
14+
15+
The [OpenTelemetry (OTel) Collector](https://opentelemetry.io/docs/collector/) acts as the entry point for all application telemetry. It receives data through the OpenTelemetry Protocol (OTLP) and routes traces and logs to Data Prepper while sending metrics to Prometheus.
16+
17+
The following example shows the key exporter and pipeline configuration for routing telemetry to Data Prepper and Prometheus:
18+
19+
```yaml
20+
exporters:
21+
otlp/opensearch:
22+
endpoint: "data-prepper:21890"
23+
tls:
24+
insecure: true
25+
26+
otlphttp/prometheus:
27+
endpoint: "http://prometheus:9090/api/v1/otlp"
28+
tls:
29+
insecure: true
30+
31+
service:
32+
pipelines:
33+
traces:
34+
receivers: [otlp]
35+
processors: [resourcedetection, memory_limiter, transform, batch]
36+
exporters: [otlp/opensearch]
37+
38+
metrics:
39+
receivers: [otlp]
40+
processors: [resourcedetection, memory_limiter, batch]
41+
exporters: [otlphttp/prometheus]
42+
43+
logs:
44+
receivers: [otlp]
45+
processors: [resourcedetection, memory_limiter, transform, batch]
46+
exporters: [otlp/opensearch]
47+
```
48+
49+
The `otlp/opensearch` exporter sends traces and logs to Data Prepper. The `otlphttp/prometheus` exporter sends metrics directly to Prometheus. For a complete OTel Collector configuration example including receivers, processors, and telemetry settings, see the [observability-stack OTel Collector config](https://github.com/opensearch-project/observability-stack/blob/main/docker-compose/otel-collector/config.yaml).
50+
51+
## Configuring Data Prepper pipelines
52+
53+
Data Prepper receives telemetry data from the OTel Collector and processes it into the formats required for APM. The pipeline architecture routes data through specialized subpipelines for log processing, trace storage, and service map generation.
54+
55+
The following example shows a complete Data Prepper pipeline configuration:
56+
57+
```yaml
58+
# Main OTLP pipeline - receives all telemetry and routes by type
59+
otlp-pipeline:
60+
source:
61+
otlp:
62+
ssl: false
63+
route:
64+
- logs: "getEventType() == \"LOG\""
65+
- traces: "getEventType() == \"TRACE\""
66+
sink:
67+
- pipeline:
68+
name: "otel-logs-pipeline"
69+
routes:
70+
- "logs"
71+
- pipeline:
72+
name: "otel-traces-pipeline"
73+
routes:
74+
- "traces"
75+
76+
# Log processing pipeline
77+
otel-logs-pipeline:
78+
workers: 5
79+
delay: 10
80+
source:
81+
pipeline:
82+
name: "otlp-pipeline"
83+
buffer:
84+
bounded_blocking:
85+
sink:
86+
- opensearch:
87+
hosts: ["https://<opensearch-host>:9200"]
88+
username: <username>
89+
password: <password>
90+
insecure: true
91+
index_type: log-analytics-plain
92+
93+
# Trace processing pipeline
94+
otel-traces-pipeline:
95+
source:
96+
pipeline:
97+
name: "otlp-pipeline"
98+
sink:
99+
- pipeline:
100+
name: "traces-raw-pipeline"
101+
- pipeline:
102+
name: "service-map-pipeline"
103+
104+
# Raw trace storage pipeline
105+
traces-raw-pipeline:
106+
source:
107+
pipeline:
108+
name: "otel-traces-pipeline"
109+
processor:
110+
- otel_traces:
111+
sink:
112+
- opensearch:
113+
hosts: ["https://<opensearch-host>:9200"]
114+
username: <username>
115+
password: <password>
116+
insecure: true
117+
index_type: trace-analytics-plain-raw
118+
119+
# Service map and APM metrics pipeline
120+
service-map-pipeline:
121+
source:
122+
pipeline:
123+
name: "otel-traces-pipeline"
124+
processor:
125+
- otel_apm_service_map:
126+
group_by_attributes: [telemetry.sdk.language] # Add any resource attribute to group by
127+
route:
128+
- otel_apm_service_map_route: 'getEventType() == "SERVICE_MAP"'
129+
- service_processed_metrics: 'getEventType() == "METRIC"'
130+
sink:
131+
- opensearch:
132+
hosts: ["https://<opensearch-host>:9200"]
133+
username: <username>
134+
password: <password>
135+
index_type: otel-v2-apm-service-map
136+
routes: [otel_apm_service_map_route]
137+
insecure: true
138+
- prometheus:
139+
url: "http://prometheus:9090/api/v1/write"
140+
routes: [service_processed_metrics]
141+
```
142+
143+
### Pipeline architecture
144+
145+
The Data Prepper pipeline processes telemetry data using the following steps:
146+
147+
1. The entry pipeline (`otlp-pipeline`) receives all telemetry and routes logs and traces to their respective subpipelines.
148+
2. The log pipeline (`otel-logs-pipeline`) writes logs to OpenSearch using the `log-analytics-plain` index type.
149+
3. The trace pipeline (`otel-traces-pipeline`) distributes traces to both the raw storage pipeline and the service map pipeline.
150+
4. The raw trace pipeline (`traces-raw-pipeline`) processes individual trace spans using the `otel_traces` processor and stores them in OpenSearch using the `trace-analytics-plain-raw` index type.
151+
5. The service map pipeline (`service-map-pipeline`) uses the `otel_apm_service_map` processor to generate service dependency maps and RED metrics. Service map topology data is written to OpenSearch, and RED metrics are exported to Prometheus through remote write.
152+
153+
Two key configuration options for the `otel_apm_service_map` processor are `group_by_attributes` (which determines how services can be grouped in the application map) and `window_duration` (which sets the time window for aggregating trace data).
154+
155+
## Verifying ingestion
156+
157+
After configuring the OTel Collector and Data Prepper, verify that data is flowing correctly:
158+
159+
1. **Verify OpenSearch indexes** - confirm the following indexes are created in your OpenSearch cluster:
160+
- `otel-v1-apm-span-*` - raw trace spans
161+
- `otel-v2-apm-service-map` - service topology data
162+
- `logs-otel-v1-*` - application logs
163+
164+
You can check indexes using:
165+
```bash
166+
curl -k -u admin:My_password_123!@# https://localhost:9200/_cat/indices?v
167+
```
168+
169+
2. **Verify Prometheus** - confirm the Data Prepper remote write target is active in your Prometheus instance by checking the Prometheus targets page at `http://localhost:9090/targets`.
170+
171+
3. **Verify in OpenSearch Dashboards** - navigate to **Observability** > **APM** to confirm that your services appear in the [Services](/docs/apm/services/) catalog and the [Application Map](/docs/apm/service-map/).
172+
173+
> **Warning:** Ensure that all port mappings are correct between the OTel Collector, Data Prepper, OpenSearch, and Prometheus. Mismatched ports are a common cause of ingestion failures.
174+
175+
## Next steps
176+
177+
- [Configuring APM in OpenSearch Dashboards](/docs/apm/configuring-apm/): Create datasets, index patterns, and configure APM settings to start using APM features.

‎docs/starlight-docs/src/content/docs/apm/index.md‎

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ title: Application Monitoring
33
description: Monitor application performance with service maps, RED metrics, and service-level views
44
sidebar:
55
label: APM
6+
order: 1
67
---
78

89
Application Monitoring gives you a real-time view of how your services are performing. It combines topology data stored in OpenSearch with time-series RED metrics (Rate, Errors, Duration) stored in Prometheus to surface health, latency, throughput, and error information across your distributed system.
@@ -46,9 +47,23 @@ From any service or operation, open correlation flyouts to jump directly to rela
4647
4. Topology and raw trace data are indexed into OpenSearch. RED metrics are exported to Prometheus via remote write.
4748
5. OpenSearch Dashboards queries both stores to render the Application Map, Services catalog, and service detail views.
4849

50+
## Configuring APM
51+
52+
To set up APM, complete the following steps:
53+
54+
1. **Create an Observability workspace** - APM features are only available within Observability workspaces. To learn how to enable and create workspaces, see [Workspace for OpenSearch Dashboards](https://opensearch.org/docs/latest/dashboards/workspace/workspace/).
55+
56+
2. **Instrument your application** - integrate [OpenTelemetry SDKs](https://opentelemetry.io/docs/instrumentation/) into your application code to generate trace and log data. See the [Send Data](/docs/send-data/opentelemetry/) section for instrumentation guides.
57+
58+
3. **Configure telemetry ingestion** - set up the OpenTelemetry Collector and Data Prepper to process and route telemetry to OpenSearch and Prometheus. See [Configuring Telemetry Ingestion](/docs/apm/configuring-telemetry-ingestion/).
59+
60+
4. **Configure APM in OpenSearch Dashboards** - create datasets, index patterns, and connect data sources in your Observability workspace. See [Setting Up APM](/docs/apm/configuring-apm/).
61+
62+
> **Note:** APM is distinct from the older [Trace analytics](https://opensearch.org/docs/latest/observing-your-data/trace/index/) and [Application analytics](https://opensearch.org/docs/latest/observing-your-data/app-analytics/) features. APM provides a more integrated experience that combines service topology, RED metrics, and in-context correlations into a single workflow.
63+
4964
## Prerequisites
5065

51-
- Data Prepper running with the trace analytics pipelines enabled (see [Application Map](/docs/apm/service-map/) for the full pipeline configuration)
66+
- Data Prepper running with the trace analytics pipelines enabled (see [Configuring Telemetry Ingestion](/docs/apm/configuring-telemetry-ingestion/) for the full pipeline configuration)
5267
- Trace data flowing via OTLP to the OTel Collector
5368
- Prometheus configured to receive remote write from Data Prepper
54-
- OpenSearch Dashboards with the Observability plugin
69+
- OpenSearch Dashboards with the Observability plugin and feature flags enabled (see [Configuring APM](#configuring-apm) above)

0 commit comments

Comments
 (0)