diff --git a/content/en/opentelemetry/integrations/runtime_metrics/_index.md b/content/en/opentelemetry/integrations/runtime_metrics/_index.md index 287427d4400..47ae6d2d3d4 100644 --- a/content/en/opentelemetry/integrations/runtime_metrics/_index.md +++ b/content/en/opentelemetry/integrations/runtime_metrics/_index.md @@ -24,6 +24,8 @@ Datadog supports OpenTelemetry runtime metrics for the following languages: - Java - .NET - Go +- NodeJS +- Python For details about host and container metrics mapping, see [OpenTelemetry Metrics Mapping][1]. @@ -51,10 +53,23 @@ If you use [OpenTelemetry manual instrumentation][4], follow the guides for your - [Java 8][5] - [Java 17][6] +#### Collector configuration + +The `jvm.gc.collections.count` and `jvm.gc.collections.elapsed` metrics require the [Delta-to-Rate Processor][8] in the OpenTelemetry Collector. Additionally, ensure that `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta` is set as well. + +```yaml +processors: + deltatorate: + metrics: + - jvm.gc.collections.count + - jvm.gc.collections.elapsed +``` + [3]: https://opentelemetry.io/docs/instrumentation/java/automatic/ [4]: https://opentelemetry.io/docs/instrumentation/java/manual/ [5]: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/runtime-telemetry/runtime-telemetry-java8/library [6]: https://github.com/open-telemetry/opentelemetry-java-instrumentation/tree/main/instrumentation/runtime-telemetry/runtime-telemetry-java17/library +[8]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatorateprocessor {{% /tab %}} @@ -62,8 +77,27 @@ If you use [OpenTelemetry manual instrumentation][4], follow the guides for your OpenTelemetry Go applications are [instrumented manually][3]. To enable runtime metrics, see the documentation for the [runtime package][4]. +#### Collector configuration + +The `process.runtime.go.gc.pause_total_ns` and `process.runtime.go.gc.count` metrics require the [Delta-to-Cumulative Processor][9] and the [Transform Processor][8] in the OpenTelemetry Collector. The Delta-to-Cumulative Processor converts delta sums to cumulative sums, and the Transform Processor converts cumulative sums to gauges. Additionally, ensure that `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta` is set to handle other metrics as well. + +```yaml +processors: + deltatocumulative: + metrics: + - process.runtime.go.gc.pause_total_ns + - process.runtime.go.gc.count + transform: + metric_statements: + - context: metric + statements: + - convert_sum_to_gauge() where name == "process.runtime.go.gc.pause_total_ns" or name == "process.runtime.go.gc.count" +``` + [3]: https://opentelemetry.io/docs/instrumentation/go/manual/ [4]: https://pkg.go.dev/go.opentelemetry.io/contrib/instrumentation/runtime +[8]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/transformprocessor +[9]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatocumulativeprocessor {{% /tab %}} @@ -87,10 +121,62 @@ The default metric export interval for the .NET OTel SDK differs from the defaul OTEL_METRIC_EXPORT_INTERVAL=10000 ``` +#### Collector configuration + +The `dotnet.process.cpu.time` metric requires the [Delta-to-Rate Processor][8] in the OpenTelemetry Collector. Additionally, ensure that `OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=delta` is set as well. + +```yaml +processors: + deltatorate: + metrics: + - dotnet.process.cpu.time +``` + [3]: https://opentelemetry.io/docs/instrumentation/net/automatic/ [4]: https://opentelemetry.io/docs/instrumentation/net/manual/ [5]: https://github.com/open-telemetry/opentelemetry-dotnet-contrib/tree/main/src/OpenTelemetry.Instrumentation.Runtime [7]: https://opentelemetry.io/docs/specs/otel/configuration/sdk-environment-variables/#periodic-exporting-metricreader +[8]: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/deltatorateprocessor + +{{% /tab %}} + +{{% tab "NodeJS" %}} + +#### Automatic instrumentation + +If you use [OpenTelemetry automatic instrumentation][3] for Node.js applications, runtime metrics are enabled by default through the [`@opentelemetry/instrumentation-runtime-node`][5] package. + +**Note**: Runtime metrics are only exported if a `MeterProvider` and metric exporter are configured. Set the `OTEL_METRICS_EXPORTER` environment variable or programmatically configure a `metricReader` in your SDK initialization. + +#### Manual instrumentation + +If you use [OpenTelemetry manual instrumentation][4], see the documentation for the [`@opentelemetry/instrumentation-runtime-node`][5] library. + +[3]: https://opentelemetry.io/docs/zero-code/js/ +[4]: https://opentelemetry.io/docs/languages/js/instrumentation/ +[5]: https://github.com/open-telemetry/opentelemetry-js-contrib/tree/main/packages/instrumentation-runtime-node + +{{% /tab %}} + +{{% tab "Python" %}} + +Runtime metrics are not enabled by default for Python applications. Install the [`opentelemetry-instrumentation-system-metrics`][5] package: + +```shell +pip install opentelemetry-instrumentation-system-metrics +``` + +If you use [automatic instrumentation][3], `opentelemetry-instrument` discovers and enables the package after installation. If you use [manual instrumentation][4], enable it in your application: + +```python +from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor + +SystemMetricsInstrumentor().instrument() +``` + +[3]: https://opentelemetry.io/docs/languages/python/automatic/ +[4]: https://opentelemetry.io/docs/languages/python/instrumentation/ +[5]: https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-system-metrics {{% /tab %}} @@ -107,132 +193,77 @@ After setup is complete, you can view runtime metrics in: ## Data collected -When using OpenTelemetry runtime metrics with Datadog, you receive both: -- Original OpenTelemetry runtime metrics -- Mapped Datadog runtime metrics for equivalent metrics +The following tables list the OpenTelemetry runtime metrics used in Datadog's out-of-the-box in-app experiences. -The OpenTelemetry runtime metrics have the following prefixes based on their source: +
OpenTelemetry runtime metrics are mapped to Datadog by metric name. Do not rename host metrics for OpenTelemetry runtime metrics as this breaks the mapping.
-| Source | Prefix | -| --- | --- | -| [OTel Collector Datadog Exporter][100] | `otel.process.runtime.*` | -| [Datadog Agent OTLP Ingest][101] | `process.runtime.*` | +{{< tabs >}} +{{< tab "Java" >}} -The following tables list the Datadog runtime metrics that are supported through OpenTelemetry mapping. "N/A" indicates that there is no OpenTelemetry equivalent metric available. +

JVM Instrumentation

+

These metrics are collected when using the latest version of the OpenTelemetry Java SDK.

+{{< mapping-table resource="jvm-instrumentation.csv">}} -
OpenTelemetry runtime metrics are mapped to Datadog by metric name. Do not rename host metrics for OpenTelemetry runtime metrics as this breaks the mapping.
+

JVM Contrib

+

These metrics are collected when using the OpenTelemetry JMX Metrics Gatherer.

+{{< mapping-table resource="jvm-contrib.csv">}} -[100]: /opentelemetry/setup/collector_exporter/ -[101]: /opentelemetry/setup/otlp_ingest_in_the_agent +

JVM Deprecated

+

These metrics are collected when using OpenTelemetry Java SDK versions 1.32.0 and earlier.

+{{< mapping-table resource="jvm-deprecated.csv">}} -{{< tabs >}} -{{% tab "Java" %}} +{{< /tab >}} -| Datadog metric | Description | OpenTelemetry metric | -| --- | --- | --- | -| `jvm.heap_memory` | The total Java heap memory used. | `process.runtime.jvm.memory.usage`
`jvm.memory.used` | -| `jvm.heap_memory_committed` | The total Java heap memory committed to be used. | `process.runtime.jvm.memory.committed`
`jvm.memory.committed` | -| `jvm.heap_memory_init` | The initial Java heap memory allocated. | `process.runtime.jvm.memory.init`
`jvm.memory.init` | -| `jvm.heap_memory_max` | The maximum Java heap memory available. | `process.runtime.jvm.memory.limit`
`jvm.memory.limit` | -| `jvm.non_heap_memory` | The total Java non-heap memory used. Non-heap memory is: `Metaspace + CompressedClassSpace + CodeCache`. | `process.runtime.jvm.memory.usage`
`jvm.memory.used` | -| `jvm.non_heap_memory_committed` | The total Java non-heap memory committed to be used. | `process.runtime.jvm.memory.committed`
`jvm.memory.committed` | -| `jvm.non_heap_memory_init` | The initial Java non-heap memory allocated. | `process.runtime.jvm.memory.init`
`jvm.memory.init` | -| `jvm.non_heap_memory_max` | The maximum Java non-heap memory available. | `process.runtime.jvm.memory.limit`
`jvm.memory.limit` | -| `jvm.gc.old_gen_size` | The current Java heap memory usage of the Old Generation memory pool. | `process.runtime.jvm.memory.usage`
`jvm.memory.used` | -| `jvm.gc.eden_size` | The current Java heap memory usage of the Eden memory pool. | `process.runtime.jvm.memory.usage`
`jvm.memory.used` | -| `jvm.gc.survivor_size` | The current Java heap memory usage of the Survivor memory pool. | `process.runtime.jvm.memory.usage`
`jvm.memory.used` | -| `jvm.gc.metaspace_size` | The current Java non-heap memory usage of the Metaspace memory pool. | `process.runtime.jvm.memory.usage`
`jvm.memory.used` | -| `jvm.thread_count` | The number of live threads. | `process.runtime.jvm.threads.count`
`jvm.thread.count` | -| `jvm.loaded_classes` | Number of classes currently loaded. | `process.runtime.jvm.classes.current_loaded`
`jvm.class.count` | -| `jvm.cpu_load.system` | Recent CPU utilization for the whole system. | `process.runtime.jvm.system.cpu.utilization`
`jvm.system.cpu.utilization` | -| `jvm.cpu_load.process` | Recent CPU utilization for the process. | `process.runtime.jvm.cpu.utilization`
`jvm.cpu.recent_utilization` | -| `jvm.buffer_pool.direct.used` | Measure of memory used by direct buffers. | `process.runtime.jvm.buffer.usage`
`jvm.buffer.memory.usage` | -| `jvm.buffer_pool.direct.count` | Number of direct buffers in the pool. | `process.runtime.jvm.buffer.count`
`jvm.buffer.count` | -| `jvm.buffer_pool.direct.limit` | Measure of total memory capacity of direct buffers. | `process.runtime.jvm.buffer.limit`
`jvm.buffer.memory.limit` | -| `jvm.buffer_pool.mapped.used` | Measure of memory used by mapped buffers. | `process.runtime.jvm.buffer.usage`
`jvm.buffer.memory.usage` | -| `jvm.buffer_pool.mapped.count` | Number of mapped buffers in the pool. | `process.runtime.jvm.buffer.count`
`jvm.buffer.count` | -| `jvm.buffer_pool.mapped.limit` | Measure of total memory capacity of mapped buffers. | `process.runtime.jvm.buffer.limit`
`jvm.buffer.memory.limit` | -| `jvm.gc.parnew.time` | The approximate accumulated garbage collection time elapsed. | N/A | -| `jvm.gc.cms.count` | The total number of garbage collections that have occurred. | N/A | -| `jvm.gc.major_collection_count` | The rate of major garbage collections. Set `new_gc_metrics: true` to receive this metric. | N/A | -| `jvm.gc.minor_collection_count` | The rate of minor garbage collections. Set `new_gc_metrics: true` to receive this metric. | N/A | -| `jvm.gc.major_collection_time` | The fraction of time spent in major garbage collection. Set `new_gc_metrics: true` to receive this metric. | N/A | -| `jvm.gc.minor_collection_time` | The fraction of time spent in minor garbage collection. Set `new_gc_metrics: true` to receive this metric. | N/A | -| `jvm.os.open_file_descriptors` | The number of open file descriptors. | N/A | +{{< tab "Go" >}} -{{% /tab %}} +{{< mapping-table resource="go-contrib-runtime.csv">}} -{{% tab "Go" %}} +{{< /tab >}} -| Datadog metric | Description | OpenTelemetry metric | -| --- | --- | --- | -| `runtime.go.num_goroutine` | Number of goroutines spawned. | `process.runtime.go.goroutines` | -| `runtime.go.num_cgo_call` | Number of CGO calls made. |`process.runtime.go.cgo.calls` | -| `runtime.go.mem_stats.lookups` | Number of pointer lookups performed by the runtime. | `process.runtime.go.mem.lookups` | -| `runtime.go.mem_stats.heap_alloc` | Bytes of allocated heap objects. | `process.runtime.go.mem.heap_alloc` | -| `runtime.go.mem_stats.heap_sys` | Bytes of heap memory obtained from the operating system. | `process.runtime.go.mem.heap_sys` | -| `runtime.go.mem_stats.heap_idle` | Bytes in idle (unused) spans. | `process.runtime.go.mem.heap_idle` | -| `runtime.go.mem_stats.heap_inuse` | Bytes in in-use spans. | `process.runtime.go.mem.heap_inuse` | -| `runtime.go.mem_stats.heap_released` | Bytes of physical memory returned to the operating system. | `process.runtime.go.mem.heap_released` | -| `runtime.go.mem_stats.heap_objects` | Number of allocated heap objects. | `process.runtime.go.mem.heap_objects` | -| `runtime.go.mem_stats.pause_total_ns` | Cumulative nanoseconds in garbage collection (GC). | `process.runtime.go.gc.pause_total_ns` | -| `runtime.go.mem_stats.num_gc` | Number of completed GC cycles. | `process.runtime.go.gc.count` | -| `runtime.go.num_cpu` | Number of CPUs detected by the runtime. | N/A | -| `runtime.go.mem_stats.alloc` | Bytes of allocated heap objects. | N/A | -| `runtime.go.mem_stats.total_alloc` | Cumulative bytes allocated for heap objects. | N/A | -| `runtime.go.mem_stats.sys` | Total bytes of memory obtained from the operating system. | N/A | -| `runtime.go.mem_stats.mallocs` | Cumulative count of heap objects allocated. | N/A | -| `runtime.go.mem_stats.frees` | Cumulative count of heap objects freed. | N/A | -| `runtime.go.mem_stats.stack_inuse` | Bytes in stack spans. | N/A | -| `runtime.go.mem_stats.stack_sys` | Bytes of stack memory obtained from the operating system. | N/A | -| `runtime.go.mem_stats.m_span_inuse` | Bytes of allocated mspan structures. | N/A | -| `runtime.go.mem_stats.m_span_sys` | Bytes of memory obtained from the operating system for mspan structures. | N/A | -| `runtime.go.mem_stats.m_cache_inuse` | Bytes of allocated mcache structures. | N/A | -| `runtime.go.mem_stats.m_cache_sys` | Bytes of memory obtained from the operating system for mcache structures. | N/A | -| `runtime.go.mem_stats.buck_hash_sys` | Bytes of memory in profiling bucket hash tables. | N/A | -| `runtime.go.mem_stats.gc_sys` | Bytes of memory in garbage collection metadata. | N/A | -| `runtime.go.mem_stats.other_sys` | Bytes of memory in miscellaneous off-heap. | N/A | -| `runtime.go.mem_stats.next_gc` | Target heap size of the next GC cycle. | N/A | -| `runtime.go.mem_stats.last_gc` | Last garbage collection finished, as nanoseconds since the UNIX epoch. | N/A | -| `runtime.go.mem_stats.num_forced_gc` | Number of GC cycles that were forced by the application calling the GC function. | N/A | -| `runtime.go.mem_stats.gc_cpu_fraction` | Fraction of this program's available CPU time used by the GC since the program started. | N/A | -| `runtime.go.gc_stats.pause_quantiles.min` | Distribution of GC pause times: minimum values. | N/A | -| `runtime.go.gc_stats.pause_quantiles.25p` | Distribution of GC pause times: 25th percentile. | N/A | -| `runtime.go.gc_stats.pause_quantiles.50p` | Distribution of GC pause times: 50th percentile. | N/A | -| `runtime.go.gc_stats.pause_quantiles.75p` | Distribution of GC pause times: 75th percentile. | N/A | -| `runtime.go.gc_stats.pause_quantiles.max` | Distribution of GC pause times: maximum values. | N/A | +{{< tab ".NET" >}} -{{% /tab %}} +

.NET System.Runtime

+

These metrics are emitted by the .NET runtime's built-in System.Runtime meter on .NET 9.0 and later. The OpenTelemetry SDK collects and exports them automatically.

+{{< mapping-table resource="dotnet.csv">}} -{{% tab ".NET" %}} +

.NET Contrib Runtime

+

These metrics are collected by the OpenTelemetry.Instrumentation.Runtime package. On .NET 9.0 and later, these overlap with the System.Runtime metrics above.

+{{< mapping-table resource="dotnet-contrib-runtime.csv">}} + +

.NET Contrib Process

+

These metrics are collected by the OpenTelemetry.Instrumentation.Process package.

+{{< mapping-table resource="dotnet-contrib-process.csv">}} + +{{< /tab >}} + +{{< tab "NodeJS" >}} + +

Node Contrib Runtime

+

These metrics are emitted from auto-instrumentation with the latest version of the OpenTelemetry Node.js SDK.

+{{< mapping-table resource="node-contrib-runtime.csv">}} + +

Node Contrib Host

+

These metrics are collected by the @opentelemetry/host-metrics package. This package is not included in OpenTelemetry automatic instrumentation and must be installed and configured manually.

+{{< mapping-table resource="node-contrib-host.csv">}} + +{{< /tab >}} + +{{% tab "Python" %}} + +The following table lists the conceptual equivalences between OpenTelemetry and Datadog Python runtime metrics. There are no direct mappings due to mismatching metric types between the two systems. -| Datadog metric | Description | OpenTelemetry metric | +| Datadog metric | Description | OpenTelemetry metric | | --- | --- | --- | -| `runtime.dotnet.threads.contention_count` | The number of times a thread stopped to wait on a lock. | `process.runtime.dotnet.`
`monitor.lock_contention.count` | -| `runtime.dotnet.exceptions.count` | The number of first-chance exceptions. | `process.runtime.dotnet.`
`exceptions.count` | -| `runtime.dotnet.gc.size.gen0` | The size of the gen 0 heap. | `process.runtime.dotnet.`
`gc.heap.size` | -| `runtime.dotnet.gc.size.gen1` | The size of the gen 1 heap. | `process.runtime.dotnet.`
`gc.heap.size` | -| `runtime.dotnet.gc.size.gen2` | The size of the gen 2 heap. | `process.runtime.dotnet.`
`gc.heap.size` | -| `runtime.dotnet.gc.size.loh` | The size of the large object heap. | `process.runtime.dotnet.`
`gc.heap.size` | -| `runtime.dotnet.gc.count.gen0` | The number of gen 0 garbage collections. | `process.runtime.dotnet.`
`gc.collections.count` | -| `runtime.dotnet.gc.count.gen1` | The number of gen 1 garbage collections. | `process.runtime.dotnet.`
`gc.collections.count` | -| `runtime.dotnet.gc.count.gen2` | The number of gen 2 garbage collections. | `process.runtime.dotnet.`
`gc.collections.count` | -| `runtime.dotnet.cpu.system` | The number of milliseconds executing in the kernel. | N/A | -| `runtime.dotnet.cpu.user` | The number of milliseconds executing outside the kernel. | N/A | -| `runtime.dotnet.cpu.percent` | The percentage of total CPU used by the application. | N/A | -| `runtime.dotnet.mem.committed` | Memory usage. | N/A | -| `runtime.dotnet.threads.count` | The number of threads. | N/A | -| `runtime.dotnet.threads.workers_count` | The number of workers in the threadpool. (.NET Core only) | N/A | -| `runtime.dotnet.threads.contention_time` | The cumulated time spent by threads waiting on a lock. (.NET Core only) | N/A | -| `runtime.dotnet.gc.memory_load` | The percentage of the total memory used by the process. The garbage collection (GC) changes its behavior when this value gets above 85. (.NET Core only) | N/A | -| `runtime.dotnet.gc.pause_time` | The amount of time the GC paused the application threads. (.NET Core only) | N/A | -| `runtime.dotnet.aspnetcore.`
`requests.total` | The total number of HTTP requests received by the server. (.NET Core only) | N/A | -| `runtime.dotnet.aspnetcore.`
`requests.failed` | The number of failed HTTP requests received by the server. (.NET Core only) | N/A | -| `runtime.dotnet.aspnetcore.`
`requests.current` | The total number of HTTP requests that have started but not yet stopped. (.NET Core only) | N/A | -| `runtime.dotnet.aspnetcore.`
`requests.queue_length` | The current length of the server HTTP request queue. (.NET Core only) | N/A | -| `runtime.dotnet.aspnetcore.`
`connections.total` | The total number of HTTP connections established to the server. (.NET Core only) | N/A | -| `runtime.dotnet.aspnetcore.`
`connections.current` | The current number of active HTTP connections to the server. (.NET Core only) | N/A | -| `runtime.dotnet.aspnetcore.`
`connections.queue_length` | The current length of the HTTP server connection queue. (.NET Core only) | N/A | +| `runtime.python.cpu.time.sys` | Number of seconds executing in the kernel. | `process.cpu.time` (`type: system`) | +| `runtime.python.cpu.time.user` | Number of seconds executing outside the kernel. | `process.cpu.time` (`type: user`) | +| `runtime.python.cpu.percent` | CPU utilization percentage. OTel divides by 100 × number of cores. | `process.cpu.utilization` | +| `runtime.python.cpu.ctx_switch.voluntary` | Number of voluntary context switches. | `process.context_switches` (`type: voluntary`) | +| `runtime.python.cpu.ctx_switch.involuntary` | Number of involuntary context switches. | `process.context_switches` (`type: involuntary`) | +| `runtime.python.gc.count.gen0` | Number of generation 0 objects. | `process.runtime.{python_implementation}.gc_count` (`count: 0`) | +| `runtime.python.gc.count.gen1` | Number of generation 1 objects. | `process.runtime.{python_implementation}.gc_count` (`count: 1`) | +| `runtime.python.gc.count.gen2` | Number of generation 2 objects. | `process.runtime.{python_implementation}.gc_count` (`count: 2`) | +| `runtime.python.mem.rss` | Resident set memory. | `process.memory.usage` | +| `runtime.python.thread_count` | Number of threads. | `process.thread.count` | {{% /tab %}}