Update setup instructions for Kafka consumer integration (DataDog#21244)

* Update setup instructions for Kafka consumer integration

* Update README.md

* Update README.md

* Update README.md

* Update kafka_consumer/README.md

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Update kafka_consumer/README.md

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* Update kafka_consumer/README.md

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

* update

---------

Co-authored-by: Bryce Eadie <bryce.eadie@datadoghq.com>

Author: piochelepiotr

kafka_consumer/README.md

@@ -20,54 +20,57 @@ The Agent's Kafka consumer check is included in the [Datadog Agent][2] package.
 ### Configuration
 
 <!-- xxx tabs xxx -->
-<!-- xxx tab "Host" xxx -->
-
-#### Host
-
-To configure this check for an Agent running on a host running your Kafka consumers:
-
-##### Metric collection
-
-1. Edit the `kafka_consumer.d/conf.yaml` file, in the `conf.d/` folder at the root of your [Agent's configuration directory][3]. See the [sample kafka_consumer.d/conf.yaml][4] for all available configuration options.
-
-2. [Restart the Agent][5].
-
-##### Log collection
-
-This check does not collect additional logs. To collect logs from Kafka brokers, see [log collection instructions for Kafka][6].
-
-<!-- xxz tab xxx -->
 <!-- xxx tab "Containerized" xxx -->
 
 #### Containerized
 
-For containerized environments, see the [Autodiscovery Integration Templates][17] for guidance on applying the parameters below.
-
-##### Metric collection
+Configure this check on a container running the Kafka Consumer.
+See the [Autodiscovery Integration Templates][17] for guidance on applying the parameters below.
+In Kubernetes, if a single consumers is running on many containers, you can setup this check as a [Cluster Check][20] to avoid having multiple checks collecting the same metrics.
 
 | Parameter            | Value                                |
 | -------------------- | ------------------------------------ |
 | `<INTEGRATION_NAME>` | `kafka_consumer`                     |
 | `<INIT_CONFIG>`      | blank or `{}`                        |
-| `<INSTANCE_CONFIG>`  | `{"kafka_connect_str": <KAFKA_CONNECT_STR>}` <br/>For example, `{"kafka_connect_str": "server:9092"}` |
+| `<INSTANCE_CONFIG>`  | `{"kafka_connect_str": "<KAFKA_CONNECT_STR>", "consumer_groups": {"<CONSUMER_NAME>": {}}}` <br/>For example, `{"kafka_connect_str": "server:9092", "consumer_groups": {"my_consumer_group": {}}}` |
 
-##### Log collection
+<!-- xxz tab xxx -->
+<!-- xxx tab "Host" xxx -->
+
+Configure this check on a host running the Kafka Consumer.
+Avoid having multiple Agents running with the same check configuration, as this puts additional pressure on your Kafka cluster.
+
+1. Edit the `kafka_consumer.d/conf.yaml` file, in the `conf.d/` folder at the root of your [Agent's configuration directory][3]. See the [sample kafka_consumer.d/conf.yaml][4] for all available configuration options. A minimal setup is:
+
+```
+instances:
+  - kafka_connect_str: <KAFKA_CONNECT_STR>
+    consumer_groups:
+      # Monitor all topics for consumer <CONSUMER_NAME>
+      <CONSUMER_NAME>: {}
+```
+
+2. [Restart the Agent][5].
 
-This check does not collect additional logs. To collect logs from Kafka brokers, see [log collection instructions for Kafka][6].
 
 <!-- xxz tab xxx -->
 <!-- xxz tabs xxx -->
 
 ### Validation
 
-[Run the Agent's status subcommand][8] and look for `kafka_consumer` under the Checks section.
+1. [Run the Agent's status subcommand][8] and look for `kafka_consumer` under the Checks section.
+2. Ensure the metric `kafka.consumer_lag` is generated for the appropriate `consumer_group`.
 
 ## Data Collected
 
 ### Metrics
 
 See [metadata.csv][9] for a list of metrics provided by this check.
 
+### Kafka messages
+
+This integration is used by [Data Streams Monitoring][18] to [retrieve messages from Kafka on demand][21].
+
 ### Events
 
 **consumer_lag**:<br>
@@ -144,3 +147,5 @@ sudo service datadog-agent restart
 [17]: https://docs.datadoghq.com/containers/kubernetes/integrations/
 [18]: /data-streams
 [19]: /integrations/kafka?search=kafka
+[20]: /containers/cluster_agent/clusterchecks/
+[21]: /data_streams/messages/