|
| 1 | +# Kafka Reporter |
| 2 | + |
| 3 | +This document describes how to configure and use the Kafka reporter in the Apache SkyWalking Go agent. The Kafka reporter provides an alternative to the default gRPC reporter, allowing you to send trace, metrics, and log data to Apache Kafka. |
| 4 | + |
| 5 | +## Overview |
| 6 | + |
| 7 | +The SkyWalking Go agent can be configured to report collected telemetry data (traces, metrics, logs) to a Kafka cluster. This is useful for scenarios where Kafka is already part of your infrastructure or when you prefer Kafka's buffering and scalability features for handling observability data. |
| 8 | + |
| 9 | +**Note:** Even when the primary data reporting is set to Kafka (`reporter.type: kafka`), the CDS functionality itself still relies on gRPC communication with the SkyWalking OAP (Observability Analysis Platform). Therefore, you **must** also configure the relevant gRPC settings under `reporter.grpc` (or their corresponding environment variables like `SW_AGENT_REPORTER_GRPC_BACKEND_SERVICE`) for CDS to work correctly. |
| 10 | + |
| 11 | + |
| 12 | +## Enabling Kafka Reporter |
| 13 | + |
| 14 | +You can enable the Kafka reporter either through environment variables or by configuring the `agent.default.yaml` file. |
| 15 | + |
| 16 | +**Using Environment Variables:** |
| 17 | + |
| 18 | +Set the `SW_AGENT_REPORTER_TYPE` environment variable to `kafka`: |
| 19 | +```bash |
| 20 | +export SW_AGENT_REPORTER_TYPE=kafka |
| 21 | +``` |
| 22 | + |
| 23 | +**Using `agent.default.yaml`:** |
| 24 | + |
| 25 | +Modify the `reporter.type` setting in your `agent.default.yaml` configuration file: |
| 26 | +```yaml |
| 27 | +reporter: |
| 28 | + type: kafka # or grpc |
| 29 | + # ... other global reporter settings |
| 30 | +``` |
| 31 | + |
| 32 | +## Configuration |
| 33 | + |
| 34 | +The Kafka reporter requires specific configurations for connecting to your Kafka cluster and specifying topics for different data types. These can be set via environment variables or in the `agent.default.yaml` file. Environment variable names generally follow the pattern `SW_AGENT_REPORTER_KAFKA_OPTION_NAME_IN_UPPERCASE`. |
| 35 | + |
| 36 | +### Core Kafka Configuration |
| 37 | + |
| 38 | +These settings are typically found under the `reporter.kafka` section in `agent.default.yaml` or can be set using the corresponding environment variables. |
| 39 | + |
| 40 | +* **Broker Addresses:** |
| 41 | + A comma-separated list of Kafka broker addresses. |
| 42 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_BROKERS` |
| 43 | + * YAML: `reporter.kafka.brokers` |
| 44 | + * Example: `kafka1:9092,kafka2:9092` |
| 45 | + |
| 46 | + |
| 47 | +* **Topic for Segments:** |
| 48 | + The Kafka topic where trace segments will be sent. |
| 49 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_TOPIC_SEGMENT` |
| 50 | + * YAML: `reporter.kafka.topic_segment` |
| 51 | + * Example: `skywalking-segments` |
| 52 | + |
| 53 | + |
| 54 | +* **Topic for Metrics:** |
| 55 | + The Kafka topic where metrics data will be sent. |
| 56 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_TOPIC_METER` |
| 57 | + * YAML: `reporter.kafka.topic_meter` |
| 58 | + * Example: `skywalking-meters` |
| 59 | + |
| 60 | + |
| 61 | +* **Topic for Logs:** |
| 62 | + The Kafka topic where log data will be sent. |
| 63 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_TOPIC_LOGGING` |
| 64 | + * YAML: `reporter.kafka.topic_logging` |
| 65 | + * Example: `skywalking-logs` |
| 66 | + |
| 67 | + |
| 68 | +* **Topic for Management:** (Optional) |
| 69 | + The Kafka topic for management-related messages (e.g., potentially for configurations or commands in future use). |
| 70 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_TOPIC_MANAGEMENT` |
| 71 | + * YAML: `reporter.kafka.topic_management` |
| 72 | + * Example: `skywalking-management` |
| 73 | + |
| 74 | + |
| 75 | +### Advanced Kafka Configuration |
| 76 | + |
| 77 | +These options allow fine-tuning of the Kafka producer behavior. They can also be configured via environment variables or in `agent.default.yaml` under `reporter.kafka`. |
| 78 | + |
| 79 | +* **Send Queue Size (Tracing, Metrics, Logs):** |
| 80 | + The maximum size of the internal queue for buffering data before sending for each data type. |
| 81 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_MAX_SEND_QUEUE` |
| 82 | + * YAML: `reporter.kafka.max_send_queue` |
| 83 | + * Default: `5000` |
| 84 | + |
| 85 | + |
| 86 | +* **Batch Size:** |
| 87 | + The maximum number of messages batched before being sent to a partition. |
| 88 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_BATCH_SIZE` |
| 89 | + * YAML: `reporter.kafka.batchSize` |
| 90 | + * Default: `1000` |
| 91 | + |
| 92 | + |
| 93 | +* **Batch Bytes:** |
| 94 | + The maximum total bytes batched before being sent to a partition. |
| 95 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_BATCH_BYTES` |
| 96 | + * YAML: `reporter.kafka.batchBytes` |
| 97 | + * Default: `1048576` (1MB) |
| 98 | + |
| 99 | + |
| 100 | +* **Batch Timeout:** |
| 101 | + The maximum time the producer will wait before sending a batch, even if `batchSize` or `batchBytes` is not met. (e.g., "1s", "500ms"). |
| 102 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_BATCH_TIMEOUT` |
| 103 | + * YAML: `reporter.kafka.batchTimeout` |
| 104 | + * Default: `1s` (1 second) |
| 105 | + |
| 106 | + |
| 107 | +* **Acknowledgement (Acks):** |
| 108 | + The level of acknowledgement required from Kafka brokers. |
| 109 | + * `0`: No acknowledgement (producer does not wait). |
| 110 | + * `1`: Leader acknowledgement (producer waits for the leader to write the record). |
| 111 | + * `-1`: All in-sync replicas acknowledgement. |
| 112 | + * Environment Variable: `SW_AGENT_REPORTER_KAFKA_ACKS` |
| 113 | + * YAML: `reporter.kafka.acks` |
| 114 | + * Default: `1` (leader) |
| 115 | + |
| 116 | + |
| 117 | +### Example `agent.default.yaml` Snippet for Kafka: |
| 118 | + |
| 119 | +```yaml |
| 120 | +reporter: |
| 121 | + # Reporter type: "grpc" or "kafka" |
| 122 | + type: ${SW_AGENT_REPORTER_TYPE:kafka} |
| 123 | + kafka: |
| 124 | + # Kafka broker addresses, comma separated |
| 125 | + brokers: ${SW_AGENT_REPORTER_KAFKA_BROKERS:127.0.0.1:9092} |
| 126 | + # Topic for segments data |
| 127 | + topic_segment: ${SW_AGENT_REPORTER_KAFKA_TOPIC_SEGMENT:skywalking-segments} |
| 128 | + # Topic for meters data |
| 129 | + topic_meter: ${SW_AGENT_REPORTER_KAFKA_TOPIC_METER:skywalking-meters} |
| 130 | + # Topic for logging data |
| 131 | + topic_logging: ${SW_AGENT_REPORTER_KAFKA_TOPIC_LOGGING:skywalking-logs} |
| 132 | + # Topic for management data |
| 133 | + topic_management: ${SW_AGENT_REPORTER_KAFKA_TOPIC_MANAGEMENT:skywalking-managements} |
| 134 | + # Send queue size |
| 135 | + max_send_queue: ${SW_AGENT_REPORTER_KAFKA_MAX_SEND_QUEUE:5000} |
| 136 | + # Batch size |
| 137 | + batch_size: ${SW_AGENT_REPORTER_KAFKA_BATCH_SIZE:1000} |
| 138 | + # Batch bytes |
| 139 | + batch_bytes: ${SW_AGENT_REPORTER_KAFKA_BATCH_BYTES:1048576} |
| 140 | + # Batch timeout millis |
| 141 | + batch_timeout_millis: ${SW_AGENT_REPORTER_KAFKA_BATCH_TIMEOUT_MILLIS:1000} |
| 142 | + # Acknowledge, 0: none, 1: leader, -1: all |
| 143 | + acks: ${SW_AGENT_REPORTER_KAFKA_ACKS:1} |
| 144 | +``` |
| 145 | +
|
| 146 | +## Data Format |
| 147 | +
|
| 148 | +The agent transforms the collected spans, metrics, and logs into the standard Apache SkyWalking data format before sending them to Kafka. The SkyWalking OAP (Observability Analysis Platform) should be configured with Kafka fetchers to consume and process this data from the specified Kafka topics. Ensure your OAP version is compatible and configured correctly to ingest data reported via Kafka. |
0 commit comments