ROB-846: update holmes docs for Loki and Tempo

docs/configuration/holmesgpt/toolsets/grafanaloki.rst

@@ -1,10 +1,10 @@
 .. _toolset_grafana_loki:
 
-Loki (Grafana)
-===============
+Loki
+====
 
-By enabling this toolset, HolmesGPT will fetch node and pods logs from `Loki <https://grafana.com/oss/loki/>`_
-by proxying through a `Grafana <https://grafana.com/oss/grafana/>`_ instance.
+By enabling this toolset, HolmesGPT will fetch node and pods logs from `Loki <https://grafana.com/oss/loki/>`_.
+Loki can be accessed directly or by proxying through a `Grafana <https://grafana.com/oss/grafana/>`_ instance.
 
 You **should** enable this toolset to replace the default :ref:`kubernetes/logs <toolset_kubernetes_logs>`
 toolset if all your kubernetes/pod logs are consolidated inside Loki. It will make it easier for HolmesGPT
@@ -13,8 +13,14 @@ to fetch incident logs, including the ability to precisely consult past logs.
 
 .. include:: ./_toolsets_that_provide_logging.inc.rst
 
+
+Proxying through Grafana
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+This is the recommended approach because we intend to add more capabilities to the toolset that are only available with Grafana.
+
 Prerequisites
-^^^^^^^^^^^^^
+-------------
 
 A `Grafana service account token <https://grafana.com/docs/grafana/latest/administration/service-accounts/>`_
 with the following permissions:
@@ -24,8 +30,7 @@ with the following permissions:
 
 Check out this `video <https://www.loom.com/share/f969ab3af509444693802254ab040791?sid=aa8b3c65-2696-4f69-ae47-bb96e8e03c47>`_ on creating a Grafana service account token.
 
-Getting Grafana URL
------------------------
+**Getting Grafana URL**
 
 You can find the Grafana URL required for Loki in your Grafana cloud account settings.
 
@@ -34,8 +39,7 @@ You can find the Grafana URL required for Loki in your Grafana cloud account set
   :align: center
 
 
-Obtaining the datasource UID
------------------------------------
+**Obtaining the datasource UID**
 
 You may have multiple Loki data sources setup in Grafana. HolmesGPT uses a single Loki datasource to
 fetch the logs and it needs to know the UID of this datasource.
@@ -74,8 +78,8 @@ A simple way to get the datasource UID is to access the Grafana API by running t
     # with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee"
 
 
-Configuration
-^^^^^^^^^^^^^
+Configuration (grafana proxy)
+-----------------------------
 
 
 .. md-tab-set::
@@ -92,9 +96,6 @@ Configuration
               api_key: <your grafana API key>
               url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
               grafana_datasource_uid: <the UID of the loki data source in Grafana>
-              labels:
-                pod: "pod"
-                namespace: "namespace"
 
           kubernetes/logs:
             enabled: false # Disable HolmesGPT's default logging mechanism
@@ -115,19 +116,102 @@ Configuration
             api_key: <your grafana API key>
             url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
             grafana_datasource_uid: <the UID of the loki data source in Grafana>
-            labels:
-                pod: "pod"
-                namespace: "namespace"
 
         kubernetes/logs:
           enabled: false # Disable HolmesGPT's default logging mechanism
 
+Direct connection
+^^^^^^^^^^^^^^^^^
+
+The toolset can directly connect to a Loki instance without proxying through a Grafana instance.
+This is done by not setting the ``grafana_datasource_uid`` field. Not setting this field makes HolmesGPT
+assume that it is directly connecting to Loki.
+
+Configuration (direct connection)
+---------------------------------
+
+.. md-tab-set::
+
+  .. md-tab-item:: Robusta Helm Chat
+
+    .. code-block:: yaml
+
+      holmes:
+        toolsets:
+          grafana/loki:
+            enabled: true
+            config:
+              url: http://loki.logging
+              headers:
+                X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled
+
+          kubernetes/logs:
+            enabled: false # Disable HolmesGPT's default logging mechanism
+
+
+    .. include:: ./_toolset_configuration.inc.rst
+
+  .. md-tab-item:: Holmes CLI
+
+    Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist:
+
+    .. code-block:: yaml
+
+      toolsets:
+        grafana/loki:
+          enabled: true
+          config:
+            url: http://loki.logging
+            headers:
+              X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled
+
+        kubernetes/logs:
+          enabled: false # Disable HolmesGPT's default logging mechanism
+
+
+Advanced configuration
+^^^^^^^^^^^^^^^^^^^^^^
 
 **Search labels**
 
 You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if your
 Loki logs settings for ``pod``, and ``namespace`` differ from the defaults in the example above.
 
+
+.. md-tab-set::
+
+  .. md-tab-item:: Robusta Helm Chat
+
+    .. code-block:: yaml
+
+      holmes:
+        toolsets:
+          grafana/loki:
+            enabled: true
+            config:
+              url: ...
+              labels:
+                  pod: "pod"
+                  namespace: "namespace"
+
+    .. include:: ./_toolset_configuration.inc.rst
+
+  .. md-tab-item:: Holmes CLI
+
+    Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist:
+
+    .. code-block:: yaml
+
+      toolsets:
+        grafana/loki:
+          enabled: true
+          config:
+            url: ...
+            labels:
+                pod: "pod"
+                namespace: "namespace"
+
+
 Use the following commands to list Loki's labels and determine which ones to use:
 
 .. code-block:: bash

docs/configuration/holmesgpt/toolsets/grafanatempo.rst

@@ -1,12 +1,18 @@
 
-Tempo (Grafana)
-================
+Tempo
+=====
 
 By enabling this toolset, HolmesGPT will be able to fetch trace information from Grafana
 Tempo to debug performance related issues, like high latency in your application.
 
+Proxying through Grafana
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+This is the recommended approach because we intend to add more capabilities to the toolset that are only available with Grafana.
+
 Prerequisites
-^^^^^^^^^^^^^
+-------------
+
 A `Grafana service account token <https://grafana.com/docs/grafana/latest/administration/service-accounts/>`_
 with the following permissions:
 
@@ -15,17 +21,15 @@ with the following permissions:
 
 Check out this `video <https://www.loom.com/share/f969ab3af509444693802254ab040791?sid=aa8b3c65-2696-4f69-ae47-bb96e8e03c47>`_ on creating a Grafana service account token.
 
-Getting Grafana URL
------------------------
+**Getting Grafana URL**
 
 You can find the Grafana URL required for Tempo in your Grafana cloud account settings.
 
 .. image:: /images/grafana-url-for-holmes.png
   :width: 600
   :align: center
 
-Obtaining the datasource UID
---------------------------------
+**Obtaining the datasource UID**
 
 You may have multiple Tempo data sources setup in Grafana. HolmesGPT uses a single Tempo datasource to
 fetch the traces and it needs to know the UID of this datasource.
@@ -60,8 +64,8 @@ A simple way to get the datasource UID is to access the Grafana API by running t
     # In this case, there is a single Tempo datasourceUID
     # with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee"
 
-Configuration
-^^^^^^^^^^^^^
+Configuration (grafana proxy)
+-----------------------------
 
 .. md-tab-set::
 
@@ -96,28 +100,113 @@ Configuration
         grafana/tempo:
           enabled: true
           config:
-              api_key: <your grafana service account token>
-              url: <your grafana url> # e.g. https://acme-corp.grafana.net
-              grafana_datasource_uid: <the UID of the tempo data source in Grafana>
-              labels:
-                pod: "k8s.pod.name"
-                namespace: "k8s.namespace.name"
-                deployment: "k8s.deployment.name"
-                node: "k8s.node.name"
-                service: "service.name"
+            api_key: <your grafana service account token>
+            url: <your grafana url> # e.g. https://acme-corp.grafana.net
+            grafana_datasource_uid: <the UID of the tempo data source in Grafana>
 
     To test, run:
 
     .. code-block:: yaml
 
         holmes ask "The payments DB is very slow, check tempo for any trace data"
 
+Direct connection
+^^^^^^^^^^^^^^^^^
+
+The toolset can directly connect to a Tempo instance without proxying through a Grafana instance.
+This is done by not setting the ``grafana_datasource_uid`` field. Not setting this field makes HolmesGPT
+assume that it is directly connecting to Tempo.
+
+Configuration (direct connection)
+---------------------------------
+
+.. md-tab-set::
+
+  .. md-tab-item:: Robusta Helm Chat
+
+    .. code-block:: yaml
+
+      holmes:
+        toolsets:
+          grafana/tempo:
+            enabled: true
+            config:
+              url: http://tempo.monitoring
+              headers:
+                X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if tempo multitenancy is enabled
+
+          kubernetes/logs:
+            enabled: false # Disable HolmesGPT's default logging mechanism
+
+
+    .. include:: ./_toolset_configuration.inc.rst
+
+  .. md-tab-item:: Holmes CLI
+
+    Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist:
+
+    .. code-block:: yaml
+
+      toolsets:
+        grafana/tempo:
+          enabled: true
+          config:
+            url: http://tempo.monitoring
+            headers:
+              X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if tempo multitenancy is enabled
+
+        kubernetes/logs:
+          enabled: false # Disable HolmesGPT's default logging mechanism
+
+
+Advanced configuration
+^^^^^^^^^^^^^^^^^^^^^^
+
 **Search labels**
 
 You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if the trace
 labels differ from the defaults in the example above.
 
 
+.. md-tab-set::
+
+  .. md-tab-item:: Robusta Helm Chat
+
+    .. code-block:: yaml
+
+      holmes:
+        toolsets:
+          grafana/tempo:
+            enabled: true
+            config:
+              url: ...
+              labels:
+                pod: "k8s.pod.name"
+                namespace: "k8s.namespace.name"
+                deployment: "k8s.deployment.name"
+                node: "k8s.node.name"
+                service: "service.name"
+
+    .. include:: ./_toolset_configuration.inc.rst
+
+  .. md-tab-item:: Holmes CLI
+
+    Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist:
+
+    .. code-block:: yaml
+
+      toolsets:
+        grafana/tempo:
+          enabled: true
+          config:
+            url: ...
+            labels:
+              pod: "k8s.pod.name"
+              namespace: "k8s.namespace.name"
+              deployment: "k8s.deployment.name"
+              node: "k8s.node.name"
+              service: "service.name"
+
 Capabilities
 ^^^^^^^^^^^^
 .. include:: ./_toolset_capabilities.inc.rst