diff --git a/docs/core_apis.md b/docs/core_apis.md index 10ee66c905..f952420b47 100644 --- a/docs/core_apis.md +++ b/docs/core_apis.md @@ -1,108 +1,5 @@ # Core Data Prepper APIs -All Data Prepper instances expose a server with some control APIs. By default, this server runs -on port 4900. Some plugins, especially Source plugins may expose other servers. These will be -on different ports and their configurations are independent of the core API. - -For example, to shut down Data Prepper, you can run: - -``` -curl -X POST http://localhost:4900/shutdown -``` - -## APIs - -The following APIs are available: - -``` -GET /list -POST /list -``` -* lists running pipelines - -``` -POST /shutdown -``` -* starts a graceful shutdown of the Data Prepper - -``` -GET /metrics/prometheus -POST /metrics/prometheus -``` -* returns a scrape of the Data Prepper metrics in Prometheus text format. This API is available provided - `metrics_registries` parameter in data prepper configuration file `data-prepper-config.yaml` has `Prometheus` as one - of the registry - -``` -GET /metrics/sys -POST /metrics/sys -``` -* returns JVM metrics in Prometheus text format. This API is available provided `metrics_registries` parameter in data - prepper configuration file `data-prepper-config.yaml` has `Prometheus` as one of the registry - -## Configuring the Server - -You can configure your Data Prepper core APIs through the `data-prepper-config.yaml` file. - -### SSL/TLS Connection - -Many of the Getting Started guides in this project disable SSL on the endpoint. - -```yaml -ssl: false -``` - -To enable SSL on your Data Prepper endpoint, configure your `data-prepper-config.yaml` -with the following: - -```yaml -ssl: true -key_store_file_path: "/usr/share/data-prepper/keystore.p12" -key_store_password: "secret" -private_key_password: "secret" -``` - -For more information on configuring your Data Prepper server with SSL, see [Server Configuration](https://github.com/opensearch-project/data-prepper/blob/main/docs/configuration.md#server-configuration). - -If you are using a self-signed certificate, you can add the `-k` flag to quickly test out sending curl requests for the core APIs with SSL. - -``` -curl -k -X POST https://localhost:4900/shutdown -``` - -### Authentication - -The Data Prepper Core APIs support HTTP Basic authentication. -You can set the username and password with the following -configuration in `data-prepper-config.yaml`: - -```yaml -authentication: - http_basic: - username: "myuser" - password: "mys3cr3t" -``` - -You can disable authentication of core endpoints using the following -configuration. Use this with caution because the shutdown API and -others will be accessible to anybody with network access to -your Data Prepper instance. - -```yaml -authentication: - unauthenticated: -``` - -### Peer Forwarder -Peer forwarder can be configured to enable stateful aggregation across multiple Data Prepper nodes. For more information on configuring Peer Forwarder, see [Peer Forwarder Configuration](https://github.com/opensearch-project/data-prepper/blob/main/docs/peer_forwarder.md). -It is supported by `service_map`, `otel_traces` and `aggregate` processors. - -### Shutdown Timeouts -When the DataPrepper `shutdown` API is invoked, the sink and processor `ExecutorService`'s are given time to gracefully shutdown and clear any in-flight data. The default graceful shutdown timeout for these `ExecutorService`'s is 10 seconds. This can be configured with the following optional parameters: - -```yaml -processor_shutdown_timeout: "PT15M" -sink_shutdown_timeout: 30s -``` - -The values for these parameters are parsed into a `Duration` object via the [DataPrepperDurationDeserializer](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-core/src/main/java/org/opensearch/dataprepper/parser/DataPrepperDurationDeserializer.java). +This has been moved to the OpenSearch Data Prepper documentation for +[Core APIs](https://docs.opensearch.org/docs/latest/data-prepper/managing-data-prepper/core-apis/) +in the **Managing OpenSearch Data Prepper** section. diff --git a/docs/log_analytics.md b/docs/log_analytics.md index 1164189b52..6446daad45 100644 --- a/docs/log_analytics.md +++ b/docs/log_analytics.md @@ -1,198 +1,5 @@ # Log Analytics -## Introduction - -Data Prepper is an extendable, configurable, and scalable solution for log ingestion into OpenSearch and Amazon OpenSearch Service. -Currently, Data Prepper is focused on receiving logs from [FluentBit](https://fluentbit.io/) via the -[Http Source](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/http-source/README.md), and processing those logs with a [Grok Processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/grok-processor/README.md) before ingesting them into OpenSearch through the [OpenSearch sink](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/README.md). - -Here is all of the components for log analytics with FluentBit, Data Prepper, and OpenSearch: -
-
-![Log Analytics Components](images/LogAnalyticsComponents.png) -
-
- -In your application environment you will have to run FluentBit. -FluentBit can be containerized through Kubernetes, Docker, or Amazon ECS. -It can also be run as an agent on EC2. -You should configure the [FluentBit http output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/http) to export log data to Data Prepper. -You will then have to deploy Data Prepper as an intermediate component and configure it to send -the enriched log data to your OpenSearch cluster or Amazon OpenSearch Service domain. From there, you can -use OpenSearch Dashboards to perform more intensive visualization and analysis. - -## Log Analytics Pipeline - -Log analytic pipelines in Data Prepper are extremely customizable. A simple pipeline is shown below. - -![](images/Log_Ingestion_FluentBit_DataPrepper_OpenSearch.jpg) - -## Http Source - -The [Http Source](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/http-source/README.md) accepts log data from FluentBit. -More specifically, this source accepts log data in a JSON array format. -This source supports industry-standard encryption in the form of TLS/HTTPS and HTTP basic authentication. - -## Processor - -The Data Prepper 1.2 release will come with a [Grok Processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/grok-processor/README.md). -The Grok Processor can be an invaluable tool to structure and extract important fields from your logs in order to make them more queryable. - -The Grok Processor comes with a wide variety of [default patterns](https://github.com/thekrakken/java-grok/blob/master/src/main/resources/patterns/patterns) that match against common log formats like apcahe logs or syslogs, -but can easily accept any custom patterns that cater to your specific log format. - -There are a lot of complex Grok features that will not be discussed here, so please read the documentation if you are interested. - -## OpenSearch sink - -We have a generic sink that writes the data to OpenSearch as the destination. The [opensearch sink](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-plugins/opensearch/README.md) has configuration options related to an OpenSearch cluster like endpoint, SSL/Username, index name, index template, index state management, etc. - -## Pipeline Configuration - -Example `pipeline.yaml` with SSL and Basic Authentication enabled for the `http-source`: - - -```yaml -log-pipeline: - source: - http: - ssl: true - ssl_certificate_file: "/full/path/to/certfile.crt" - ssl_key_file: "/full/path/to/keyfile.key" - # The default port that will listen for incoming logs - port: 2021 - authentication: - http_basic: - username: "myuser" - password: "mys3cret" - processor: - - grok: - match: - # This will match logs with a "log" key against the COMMONAPACHELOG pattern (ex: { "log": "actual apache log..." } ) - # You should change this to match what your logs look like. See the grok documenation to get started. - log: [ "%{COMMONAPACHELOG}" ] - sink: - - opensearch: - hosts: [ "https://localhost:9200" ] - # Change to your credentials - username: "admin" - password: "admin" - # Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate - #cert: /path/to/cert - # If you are connecting to an Amazon OpenSearch Service domain without - # Fine-Grained Access Control, enable these settings. Comment out the - # username and password above. - #aws_sigv4: true - #aws_region: us-east-1 - # Since we are grok matching for apache logs, it makes sense to send them to an OpenSearch index named apache_logs. - # You should change this to correspond with how your OpenSearch indices are set up. - index: apache_logs -``` -

-Example `pipeline.yaml` without SSL and Basic Authentication enabled for the `http-source`: - -```yaml -log-pipeline: - source: - http: - # Explicitly disable SSL - ssl: false - # Explicitly disable authentication - authentication: - unauthenticated: - # The default port that will listen for incoming logs - port: 2021 - processor: - - grok: - match: - # This will match logs with a "log" key against the COMMONAPACHELOG pattern (ex: { "log": "actual apache log..." } ) - # You should change this to match what your logs look like. See the grok documenation to get started. - log: [ "%{COMMONAPACHELOG}" ] - sink: - - opensearch: - hosts: [ "https://localhost:9200" ] - # Change to your credentials - username: "admin" - password: "" - # Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate - #cert: /path/to/cert - # If you are connecting to an Amazon OpenSearch Service domain without - # Fine-Grained Access Control, enable these settings. Comment out the - # username and password above. - #aws_sigv4: true - #aws_region: us-east-1 - # Since we are grok matching for apache logs, it makes sense to send them to an OpenSearch index named apache_logs. - # You should change this to correspond with how your OpenSearch indices are set up. - index: apache_logs -``` - -This pipeline configuration is an example of apache log ingestion. Don't forget that you can easily configure the Grok Processor for your own custom logs. - -You will need to modify the configuration above for your OpenSearch cluster. - -The main changes you will need to make are: - -* `hosts` - Set to your hosts -* `index` - Change this to the OpenSearch index you want to send logs to -* `username`- Provide the OpenSearch username -* `password` - Provide your OpenSearch password -* `aws_sigv4` - If you use Amazon OpenSearch Service with AWS signing, set this to true. It will sign requests with the default AWS credentials provider. -* `aws_region` - If you use Amazon OpenSearch Service with AWS signing, set this value to your region. -## FluentBit - -You will have to run FluentBit in your service environment. You can find the installation guide of FluentBit [here](https://docs.fluentbit.io/manual/installation/getting-started-with-fluent-bit). -Please ensure that you can configure the [FluentBit http output plugin](https://docs.fluentbit.io/manual/pipeline/outputs/http) to your Data Prepper Http Source. Below is an example `fluent-bit.conf` that tails a log file named `test.log` and forwards it to a locally running Data Prepper's http source, which runs -by default on port 2021. Note that you should adjust the file `path`, output `Host` and `Port` according to how and where you have FluentBit and Data Prepper running. - -Example `fluent-bit.conf` without SSL and Basic Authentication enabled on the http source: - -``` -[INPUT] - name tail - refresh_interval 5 - path test.log - read_from_head true - -[OUTPUT] - Name http - Match * - Host localhost - Port 2021 - URI /log/ingest - Format json -``` - -If your http source has SSL and Basic Authentication enabled, you will need to add the details -of `http_User`, `http_Passwd`, `tls.crt_file`, and `tls.key_file` to the `fluent-bit.conf` as shown below. - -Example `fluent-bit.conf` with SSL and Basic Authentication enabled on the http source: - -``` -[INPUT] - name tail - refresh_interval 5 - path test.log - read_from_head true - -[OUTPUT] - Name http - Match * - Host localhost - http_User myuser - http_Passwd mys3cret - tls On - tls.crt_file /full/path/to/certfile.crt - tls.key_file /full/path/to/keyfile.key - Port 2021 - URI /log/ingest - Format json -``` - -## Next Steps - -Follow the [Log Ingestion Demo Guide](../examples/log-ingestion/README.md) to go through a specific example of Apache log ingestion from `FluentBit -> Data Prepper -> OpenSearch` running through Docker. - -In the future, Data Prepper will contain additional sources and processors which will make more complex log analytic pipelines available. Check out our [Roadmap](https://github.com/opensearch-project/data-prepper/projects/1) to see what is coming. - -If there is a specifc source, processor, or sink that you would like to include in your log analytic workflow, and it is not currently on the Roadmap, please bring it to our attention by making a Github issue. Additionally, if you -are interested in contributing, see our [Contribuing Guidelines](../CONTRIBUTING.md) as well as our [Developer Guide](developer_guide.md) and [Plugin Development Guide](plugin_development.md). +This has been moved to the OpenSearch Data Prepper documentation for +[Log Analytics](https://docs.opensearch.org/docs/latest/data-prepper/common-use-cases/log-analytics/) +in the **Common use cases** section. diff --git a/docs/migrating_from_opendistro.md b/docs/migrating_from_opendistro.md index e4e6e9bc75..3ecedf2b19 100644 --- a/docs/migrating_from_opendistro.md +++ b/docs/migrating_from_opendistro.md @@ -1,25 +1,4 @@ # Migrating from Open Distro Data Prepper -Starting with Data Prepper 1.1, there is only one distribution of -Data Prepper - Open Search Data Prepper. This document is here to help existing users migrate -from the old Open Distro Data Prepper to OpenSearch Data Prepper. - - -### Change your Pipeline Configuration - -The `elasticsearch` sink has changed to `opensearch`. You will -need to change your existing pipeline to use the `opensearch` plugin -instead of `elasticsearch`. - -Please note that while the plugin is titled `opensearch` it remains compatible -with Open Distro and ElasticSearch 7.x. - -### Update Docker Image - -The Open Distro Data Prepper Docker image was located at `amazon/opendistro-for-elasticsearch-data-prepper`. -You will need to change this value to `opensearchproject/data-prepper`. - -## More Information - -You can find more information about Data Prepper configurations -by going to the [Getting Started](getting_started.md) guide. +This has been moved to the OpenSearch Data Prepper documentation for +[Migrating from Open Distro](https://docs.opensearch.org/docs/latest/data-prepper/migrate-open-distro/). diff --git a/docs/peer_forwarder.md b/docs/peer_forwarder.md index e1d4ee33ac..e867681a4a 100644 --- a/docs/peer_forwarder.md +++ b/docs/peer_forwarder.md @@ -1,155 +1,5 @@ - ## Peer Forwarder -An HTTP service which performs peer forwarding of `Event` between Data Prepper nodes for aggregation. Currently, supported by `aggregate`, `service_map`, `otel_traces` processors. - -Peer Forwarder groups events based on the identification keys provided the processors. -For `service_map` and `otel_traces` it's `traceId` by default and can not be configured. -It's configurable for `aggregate` processor using `identification_keys` configuration option. You can find more information about identification keys [here](https://github.com/opensearch-project/data-prepper/tree/main/data-prepper-plugins/aggregate-processor#identification_keys). - ---- - -Presently peer discovery is provided by either a static list or by a DNS record lookup or AWS Cloudmap. - -### Static discovery mode -Static discover mode allows Data Prepper node to discover nodes using a list of IP addresses or domain names. -```yaml -peer_forwarder: - discovery_mode: static - static_endpoints: ["data-prepper1", "data-prepper2"] -``` - -### DNS lookup discovery mode -We recommend using DNS discovery over static discovery when scaling out a Data Prepper cluster. The core concept is to configure a DNS provider to return a list of Data Prepper hosts when given a single domain name. -This is a [DNS A Record](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) which indicates a list of IP addresses of a given domain. -```yaml -peer_forwarder: - discovery_mode: dns - domain_name: "data-prepper-cluster.my-domain.net" -``` - -### AWS Cloud Map discovery mode - -[AWS Cloud Map](https://docs.aws.amazon.com/cloud-map/latest/dg/what-is-cloud-map.html) provides API-based service discovery as well as DNS-based service discovery. - -Peer forwarder can use the API-based service discovery. To support this you must have an existing -namespace configured for API instance discovery. You can create a new one following the instructions -provided by the [Cloud Map documentation](https://docs.aws.amazon.com/cloud-map/latest/dg/working-with-namespaces.html). - -Your Data Prepper configuration needs to include: -* `aws_cloud_map_namespace_name` - Set to your Cloud Map Namespace name -* `aws_cloud_map_service_name` - Set to the service name within your specified Namespace -* `aws_region` - The AWS region where your namespace exists. -* `discovery_mode` - Set to `aws_cloud_map` - -Your Data Prepper configuration can optionally include: -* `aws_cloud_map_query_parameters` - Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. - -Example configuration: - -```yaml -peer_forwarder: - discovery_mode: aws_cloud_map - aws_cloud_map_namespace_name: "my-namespace" - aws_cloud_map_service_name: "data-prepper-cluster" - aws_cloud_map_query_parameters: - instance_type: "r5.xlarge" - aws_region: "us-east-1" -``` - -The Data Prepper must also be running with the necessary permissions. The following -IAM policy shows the necessary permissions. - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "CloudMapPeerForwarder", - "Effect": "Allow", - "Action": "servicediscovery:DiscoverInstances", - "Resource": "*" - } - ] -} -``` ---- -## Configuration - -* `port`(Optional): An `int` between 0 and 65535 represents the port peer forwarder server is running on. Default value is `4994`. -* `request_timeout`(Optional): Duration - An `int` representing the request timeout in milliseconds for Peer Forwarder HTTP server. Default value is `10000`. -* `server_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder server. Defaults to `200`. -* `client_thread_count`(Optional): An `int` representing number of threads used by Peer Forwarder client. Defaults to `200`. -* `maxConnectionCount`(Optional): An `int` representing maximum number of open connections for Peer Forwarder server. Default value is `500`. -* `discovery_mode`(Optional): A `String` representing the peer discovery mode to be used. Allowable values are `local_node`, `static`, `dns`, and `aws_cloud_map`. Defaults to `local_node` which processes events locally. -* `static_endpoints`(Optional): A `list` containing endpoints of all Data Prepper instances. Required if `discovery_mode` is set to `static`. -* `domain_name`(Optional): A `String` representing single domain name to query DNS against. Typically, used by creating multiple [DNS A Records](https://www.cloudflare.com/learning/dns/dns-records/dns-a-record/) for the same domain. Required if `discovery_mode` is set to `dns`. -* `aws_cloud_map_namespace_name`(Optional) - A `String` representing the Cloud Map namespace when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_service_name`(Optional) - A `String` representing the Cloud Map service when using AWS Cloud Map service discovery. Required if `discovery_mode` is set to `aws_cloud_map`. -* `aws_cloud_map_query_parameters`(Optional): A `Map` of Key/value pairs to filter the results based on the custom attributes attached to an instance. Only instances that match all the specified key-value pairs are returned. -* `buffer_size`(Optional): An `int` representing max number of unchecked records the buffer accepts (num of unchecked records = num of records written into the buffer + num of in-flight records not yet checked by the Checkpointing API). Default is `512`. -* `batch_size`(Optional): An `int` representing max number of records the buffer returns on read. Default is `48`. -* `batch_delay`(Optional): An `int` representing the maximum duration in milliseconds to retrieve `batch_size` records from the peer forwarder buffer. If the `batch_size` has not been reached before this duration is exceeded, a partial batch is used. If this value is set to 0, all available records up to the batch size will be immediately returned. If the buffer is empty, the buffer will block for up to 5 milliseconds to wait for records. Default value is `3000`. -* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. -* `drain_timeout`(Optional) : A `Duration` representing the wait time for the peer forwarder to complete processing data before shutdown. -* `forwarding_batch_size`(Optional) : An `int` representing the maximum number of records to send in each request to a peer. Default value is `1500`, maximum value is `15000`. -* `forwarding_batch_queue_depth`(Optional) : An `int` representing the depth of the batching queue. This value is a scalar used to determine the size of the LinkedBlockingQueues used for batching records before they are sent to a peer. The queue size is determined by the formula: `workers` * `forwarding_batch_size` * `forwarding_batch_queue_depth`. Default value is `1`. -* `forwarding_batch_timeout`(Optional) : A `Duration` representing the maximum time that can occur between flushing batches to a peer. Default is `3s`. - -### SSL -The SSL configuration for setting up trust manager for peer forwarding client to connect to other Data Prepper instances. - -* `ssl`(Optional) : A `boolean` that enables TLS/SSL. Default value is `true`. -* `ssl_certificate_file`(Optional) : A `String` representing the SSL certificate chain file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_certificate.pem` which is default certificate file. Read more about how the certificate file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_key_file`(Optional) : A `String` represents the SSL key file path or AWS S3 path. S3 path example `s3:///`. Defaults to `config/default_private_key.pem` which is default private key file. Read more about how the private key file is generated [here](https://github.com/opensearch-project/data-prepper/tree/main/examples/certificates). -* `ssl_insecure_disable_verification`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain. Default value is `false`. -* `ssl_fingerprint_verification_only`(Optional) : A `boolean` that disables the verification of server's TLS certificate chain and instead verifies only the certificate fingerprint. Default value is `false`. -* `use_acm_certificate_for_ssl`(Optional) : A `boolean` that enables TLS/SSL using certificate and private key from AWS Certificate Manager (ACM). Default is `false`. -* `acm_certificate_arn`(Optional) : A `String` represents the ACM certificate ARN. ACM certificate take preference over S3 or local file system certificate. Required if `use_acm_certificate_for_ssl` is set to `true`. -* `acm_private_key_password`(Optional) : A `String` that represents the ACM private key password which that will be used to decrypt the private key. If it's not provided, a random password will be generated. -* `acm_certificate_timeout_millis`(Optional) : An `int` representing the timeout in milliseconds for ACM to get certificates. Default value is `120000`. -* `aws_region`(Optional) : A `String` represents the AWS region to use `ACM`, `S3` or `AWS Cloud Map`. Required if `use_acm_certificate_for_ssl` is set to `true` or `ssl_certificate_file` and `ssl_key_file` is `AWS S3` path or if `discovery_mode` is set to `aws_cloud_map`. - -```yaml -peer_forwarder: - ssl: true - ssl_certificate_file: "" - ssl_key_file: "" -``` - -### Authentication - -* `authentication`(Optional) : A `Map` that enables mTLS. It can either be `mutual_tls` or `unauthenticated`. Default value is `unauthenticated`. -```yaml -peer_forwarder: - authentication: - mutual_tls: -``` - -## Metrics - -Core Peer Forwarder introduces the following custom metrics and all the metrics are prefixed by `core.peerForwarder` - -### Timer - -- `requestForwardingLatency`: measures latency of forwarding requests by peer forwarder client. -- `requestProcessingLatency`: measures latency of processing requests by peer forwarder server. - -### Counter - -- `requests`: measures total number of forwarded requests. -- `requestsFailed`: measures total number of failed requests. Requests with HTTP response code other than `200`. -- `requestsSuccessful`: measures total number of successful requests. Requests with HTTP response code `200`. -- `requestsTooLarge`: measures total number of requests which are too large to be written to peer forwarder buffer. Requests with HTTP response code `413`. -- `requestTimeouts`: measures total number of requests which timed out while writing content to peer forwarder buffer. Requests with HTTP response code `408`. -- `requestsUnprocessable`: measures total number of requests which failed due to unprocessable entity. Requests with HTTP response code `422`. -- `badRequests`: measures total number of requests with bad request format. Requests with HTTP response code `400`. -- `recordsSuccessfullyForwarded`: measures total number of forwarded records successfully. -- `recordsFailedForwarding`: measures total number of records failed to be forwarded. -- `recordsToBeForwarded`: measures total number of records to be forwarded. -- `recordsToBeProcessedLocally`: measures total number of records to be processed locally. -- `recordsActuallyProcessedLocally`: measures total number of records actually processed locally. Sum of `recordsToBeProcessedLocally` and `recordsFailedForwarding`. -- `recordsReceivedFromPeers`: measures total number of records received from remote peers. - -### Gauge -- `peerEndpoints`: measures number of dynamically discovered peer data-prepper endpoints. For `static` mode, the size is fixed. +This has been moved to the OpenSearch Data Prepper documentation for +[Peer Forwarder](https://docs.opensearch.org/docs/latest/data-prepper/managing-data-prepper/peer-forwarder/) +in the **Managing OpenSearch Data Prepper** section. diff --git a/docs/trace_analytics.md b/docs/trace_analytics.md index a7cfa5df16..b6d4a6cca7 100644 --- a/docs/trace_analytics.md +++ b/docs/trace_analytics.md @@ -1,258 +1,5 @@ # Trace Analytics -## Introduction - -Data Prepper ingests [Trace Analytics](https://opensearch.org/docs/latest/observability-plugin/trace/index/) into OpenSearch -and Amazon OpenSearch Service. Data Prepper is a last mile server-side component which collects telemetry data -from [AWS Distro OpenTelemetry collector](https://aws-otel.github.io/docs/getting-started/collector) or -[OpenTelemetry collector](https://github.com/open-telemetry/opentelemetry-collector) and transforms it for OpenSearch. -The transformed trace data is the visualized using the -[Trace Analytics OpenSearch Dashboards plugin](https://opensearch.org/docs/latest/observability-plugin/trace/ta-dashboards/), -which provides at-a-glance visibility into your application performance, along with the ability to drill down on individual traces. - -Here is how all the components work in trace analytics: -
-
-![Trace Analytics Pipeline](images/TraceAnalyticsComponents.png) -
-
- -In your service environment you will have to run OpenTelemetry Collector. You can run it as a sidecar or daemonset for EKS, a sidecar for ECS, or an agent on EC2. You should configure the collector to export trace data to Data Prepper. You will then have to deploy Data Prepper as an intermediate component and configure it to send the enriched trace data to your OpenSearch cluster or Amazon OpenSearch Service domain. Then using OpenSearch Dashboards you can visualize and detect problems in your distributed applications. - -## Trace Analytics Pipeline - -To achieve trace analytics in Data Prepper we have three pipelines `otel-trace-pipeline`, `raw-trace-pipeline` and `service-map-pipeline` - -![Trace Analytics Pipeline](images/TraceAnalyticsFeature.jpg) - - -### OpenTelemetry Trace Source - -The [OpenTelemetry source](../data-prepper-plugins/otel-trace-source/README.md) accepts trace data from the OpenTelemetry collector. The source depends on [OpenTelemetry Protocol](https://github.com/open-telemetry/opentelemetry-specification/tree/master/specification/protocol). The source officially support transport over gRPC. The source also supports industry-standard encryption (TLS/HTTPS). - -### Processor - -We have two processor for the Trace Analytics feature, -* *otel_traces* - This is a processor that receives collection of [Span](../data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records sent from [otel-trace-source](../data-prepper-plugins/otel-trace-source/README.md), does stateful processing on extracting and filling-in trace group related fields. -* *otel_trace_group* - This is a processor that fills in the missing trace group related fields in the collection of [Span](../data-prepper-api/src/main/java/org/opensearch/dataprepper/model/trace/Span.java) records by looking up the opensearch backend. -* *service_map* - This processor performs the required preprocessing on the trace data and build metadata to display the service-map OpenSearch Dashboards dashboards. - - -### OpenSearch sink - -We have a generic sink that writes the data to OpenSearch as the destination. The [opensearch sink](../data-prepper-plugins/opensearch/README.md) has configuration options related to OpenSearch cluster like endpoint, SSL/Username, index name, index template, index state management, etc. -For the trace analytics feature, the sink has specific configurations which enables the sink to use indices and index templates specific to this feature. Trace analytics specific OpenSearch indices are, - -* *otel-v1-apm-span* - This index stores the output from [otel-trace-raw-processor](../data-prepper-plugins/otel-trace-raw-processor/README.md). -* *otel-v1-apm-service-map* - This index stores the output from the [service-map-processor](../data-prepper-plugins/service-map-stateful/README.md). - -### Pipeline Configuration - -Example `otel-trace-source` with SSL and Basic Authentication enabled. Note that you will have to change your `otel-collector-config.yaml` accordingly: - -```yaml -source: - otel_trace_source: - #record_type: event # Add this when using Data Prepper 1.x. This option is removed in 2.0 - ssl: true - sslKeyCertChainFile: "/full/path/to/certfile.crt" - sslKeyFile: "/full/path/to/keyfile.key" - authentication: - http_basic: - username: "my-user" - password: "my_s3cr3t" -``` - - -Example `pipeline.yaml` without SSL and Basic Authentication for the `otel-trace-source`: - -```yaml -otel-trace-pipeline: - # workers is the number of threads processing data in each pipeline. - # We recommend same value for all pipelines. - # default value is 1, set a value based on the machine you are running Data Prepper - workers: 8 - # delay in milliseconds is how often the worker threads should process data. - # Recommend not to change this config as we want the otel-trace-pipeline to process as quick as possible - # default value is 3_000 ms - delay: "100" - source: - otel_trace_source: - #record_type: event # Add this when using Data Prepper 1.x. This option is removed in 2.0 - ssl: false # Change this to enable encryption in transit - authentication: - unauthenticated: - buffer: - bounded_blocking: - # buffer_size is the number of ExportTraceRequest from otel-collector the data prepper should hold in memeory. - # We recommend to keep the same buffer_size for all pipelines. - # Make sure you configure sufficient heap - # default value is 12800 - buffer_size: 25600 - # This is the maximum number of request each worker thread will process within the delay. - # Default is 200. - # Make sure buffer_size >= workers * batch_size - batch_size: 400 - sink: - - pipeline: - name: "raw-pipeline" - - pipeline: - name: "service-map-pipeline" -raw-pipeline: - # Configure same as the otel-trace-pipeline - workers: 8 - # We recommend using the default value for the raw-pipeline. - delay: "3000" - source: - pipeline: - name: "otel-trace-pipeline" - buffer: - bounded_blocking: - # Configure the same value as in otel-trace-pipeline - # Make sure you configure sufficient heap - # default value is 12800 - buffer_size: 25600 - # This is the maximum number of request each worker thread will process within the delay. - # Default is 200. - # Make sure buffer_size >= workers * batch_size - # The raw processor does bulk request to your OpenSearch sink, so configure the batch_size higher. - batch_size: 3200 - processor: - - otel_traces: - - otel_trace_group: - hosts: [ "https://localhost:9200" ] - # Change to your credentials - username: "admin" - password: "" - # Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate - #cert: /path/to/cert - # If you are connecting to an Amazon OpenSearch Service domain without - # Fine-Grained Access Control, enable these settings. Comment out the - # username and password above. - #aws_sigv4: true - #aws_region: us-east-1 - sink: - - opensearch: - hosts: [ "https://localhost:9200" ] - index_type: trace-analytics-raw - # Change to your credentials - username: "admin" - password: "" - # Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate - #cert: /path/to/cert - # If you are connecting to an Amazon OpenSearch Service domain without - # Fine-Grained Access Control, enable these settings. Comment out the - # username and password above. - #aws_sigv4: true - #aws_region: us-east-1 -service-map-pipeline: - workers: 8 - delay: "100" - source: - pipeline: - name: "otel-trace-pipeline" - processor: - - service_map: - # The window duration is the maximum length of time the data prepper stores the most recent trace data to evaluvate service-map relationships. - # The default is 3 minutes, this means we can detect relationships between services from spans reported in last 3 minutes. - # Set higher value if your applications have higher latency. - window_duration: 180 - buffer: - bounded_blocking: - # buffer_size is the number of ExportTraceRequest from otel-collector the data prepper should hold in memeory. - # We recommend to keep the same buffer_size for all pipelines. - # Make sure you configure sufficient heap - # default value is 12800 - buffer_size: 25600 - # This is the maximum number of request each worker thread will process within the delay. - # Default is 200. - # Make sure buffer_size >= workers * batch_size - batch_size: 400 - sink: - - opensearch: - hosts: [ "https://localhost:9200" ] - index_type: trace-analytics-service-map - # Change to your credentials - username: "admin" - password: "" - # Add a certificate file if you are accessing an OpenSearch cluster with a self-signed certificate - #cert: /path/to/cert - # If you are connecting to an Amazon OpenSearch Service domain without - # Fine-Grained Access Control, enable these settings. Comment out the - # username and password above. - #aws_sigv4: true - #aws_region: us-east-1 -``` - -You will need to modify the configuration above for your OpenSearch cluster. Note that it has two -`opensearch` sinks which need to be modified. - -The main changes you will need to make are: -* `hosts` - Set to your hosts -* `username` - Provide the OpenSearch username -* `password` - Provide your OpenSearch password -* `aws_sigv4` - If you are Amazon OpenSearch Service with AWS signing, set this to `true`. It will sign requests with the default AWS credentials provider. -* `aws_region` - If you are Amazon OpenSearch Service with AWS signing, set this value to your region. - -The the [Data Prepper OpenSearch Sink](../data-prepper-plugins/opensearch/README.md#Configuration) documents -other configurations available for OpenSearch. - -## OpenTelemetry Collector - -You will have to run OpenTelemetry collector in your service environment. You can find the installation guide of OpenTelemetry collector [here](https://opentelemetry.io/docs/collector/getting-started/#getting-started). Please ensure you that you configure the collector with an exporter configured to your Data Prepper. Below is an example `otel-collector-config.yaml` that receives data from various instrumentations and export it to Data Prepper. - -``` -receivers: - jaeger: - protocols: - grpc: - otlp: - protocols: - grpc: - zipkin: - -processors: - batch/traces: - timeout: 1s - send_batch_size: 50 - -exporters: - otlp/data-prepper: - endpoint: localhost:21890 - tls: - insecure: true - -service: - pipelines: - traces: - receivers: [jaeger, otlp, zipkin] - processors: [batch/traces] - exporters: [otlp/data-prepper] -``` - -Configure your application to use the OpenTelemetry Collector. - -You will normally run the OpenTelemetry Collector alongside your application. - -## Next Steps and More Information - -The [OpenSearch Trace Analytics](https://opensearch.org/docs/monitoring-plugins/trace/index/) documentation -provides additional details on configuring OpenSearch for viewing trace analytics. In particular, it -documents how to use OpenSearch Dashboards. - -The [Trace Tuning page](trace_tuning.md) has information to help you tune and scale Data Prepper for -trace analytics use cases. - -## Migrating to Data Prepper 2.0 - -Starting in Data Prepper 1.4, the trace processing uses Data Prepper's Event model. This allows -pipeline authors the ability to configure other processors to modify spans or traces. - -To provide a migration path, Data Prepper 1.4 introduced the following changes. -* The `otel_trace_source` has an optional parameter `record_type` which can be set to `event`. When configured, it will output event objects. -* The `otel_traces` replaces `otel_trace_raw_prepper` for event-based spans. -* The `otel_trace_group` replaces `otel_trace_group_prepper` for event-based spans. - -In Data Prepper 2.0, the `otel_trace_source` will only output Events. Data Prepper 2.0 also removes -`otel_trace_raw_prepper` and `otel_trace_group_prepper` entirely. To help migrate to 2.0, -you can configure your trace pipeline using the Event model. +This has been moved to the OpenSearch Data Prepper documentation for +[Trace Analytics](https://docs.opensearch.org/docs/latest/data-prepper/common-use-cases/trace-analytics/) +in the **Common use cases** section.