ROB-708: use loki as fully fledged log provider (#1746)

nherment · web-flow · commit 92a771c04bfc · 2025-03-18T08:29:26.000Z
* feat: update loki and tempo docs

* feat: add docs for datetime toolset

* feat: add docs for datetime toolset

* doc: update grafana loki config
diff --git a/docs/configuration/holmesgpt/builtin_toolsets.rst b/docs/configuration/holmesgpt/builtin_toolsets.rst
@@ -62,9 +62,14 @@ by the user by providing credentials or API keys to external systems.
         :link: toolsets/docker
         :link-type: doc
 
-    .. grid-item-card:: :octicon:`cpu;1em;` Grafana
+    .. grid-item-card:: :octicon:`cpu;1em;` Grafana Loki
         :class-card: sd-bg-light sd-bg-text-light
-        :link: toolsets/grafana
+        :link: toolsets/grafanaloki
+        :link-type: doc
+
+    .. grid-item-card:: :octicon:`cpu;1em;` Grafana Tempo
+        :class-card: sd-bg-light sd-bg-text-light
+        :link: toolsets/grafanatempo
         :link-type: doc
 
     .. grid-item-card:: :octicon:`cpu;1em;` Helm
diff --git a/docs/configuration/holmesgpt/toolsets/datetime.rst b/docs/configuration/holmesgpt/toolsets/datetime.rst
@@ -3,9 +3,13 @@ Datetime :checkmark:`_`
 .. include:: ./_toolset_enabled_by_default.inc.rst
 
 By enabling this toolset, HolmesGPT will be able to get the current UTC date and time.
-This feature works well with other toolsets. For example, the :doc:`prometheus <prometheus>`
-toolset needs ``start`` and ``end`` time parameters to properly create and execute
-PromQL queries.
+This feature should be kept enabled as it can be necessary for other toolsets that rely
+on dates and time.
+
+The following built-in toolsets depend on ``datetime``:
+
+* :ref:`grafana/loki <toolset_grafana_loki>`
+* :ref:`prometheus/metrics <toolset_prometheus>`
 
 Configuration
 -------------
diff --git a/docs/configuration/holmesgpt/toolsets/grafanaloki.rst b/docs/configuration/holmesgpt/toolsets/grafanaloki.rst
@@ -1,13 +1,18 @@
+.. _toolset_grafana_loki:
+
 Loki (Grafana)
 ===============
 
-By enabling this toolset, HolmesGPT will be able to query logs from `Loki <https://grafana.com/oss/loki/>`_
+By enabling this toolset, HolmesGPT will fetch node and pods logs from `Loki <https://grafana.com/oss/loki/>`_
 by proxying through a `Grafana <https://grafana.com/oss/grafana/>`_ instance.
 
-After this toolset is enabled, you can ask Holmes questions like: *Look at my cluster with kubectl, pick an arbitrary node, then fetch logs from loki for that node. Summarize all problems you find.*
+You **should** enable this toolset to replace the default :ref:`kubernetes/logs <toolset_kubernetes_logs>`
+toolset if all your kubernetes/pod logs are consolidated inside Loki. It will make it easier for HolmesGPT
+to fetch incident logs, including the ability to precisely consult past logs.
 
 Prerequisites
 ^^^^^^^^^^^^^
+
 A `Grafana service account token <https://grafana.com/docs/grafana/latest/administration/service-accounts/>`_
 with the following permissions:
 
@@ -19,48 +24,80 @@ Check out this `video <https://www.loom.com/share/f969ab3af509444693802254ab0407
 Getting Grafana URL
 -----------------------
 
-You can find the Grafana URL required for Loki and Tempo in your Grafana cloud account settings. 
+You can find the Grafana URL required for Loki in your Grafana cloud account settings.
 
 .. image:: /images/grafana-url-for-holmes.png
   :width: 600
   :align: center
 
+
+Obtaining the datasource UID
+-----------------------
+
+You may have multiple Loki data sources setup in Grafana. HolmesGPT uses a single Loki datasource to
+fetch the logs and it needs to know the UID of this datasource.
+
+A simple way to get the datasource UID is to access the Grafana API by running the following request:
+
+.. code-block:: bash
+
+    # port forward if you are using Robusta's grafana from your kubernetes cluster
+    kubectl port-forward svc/robusta-grafana 3000:80
+
+    # List the loki data sources
+    curl -s -u <username>:<password> http://localhost:3000/api/datasources | jq '.[] | select(.type == "loki")'
+    {
+        "id": 2,
+        "uid": "klja8hsa-8a9c-4b35-1230-7baab22b02ee",
+        "orgId": 1,
+        "name": "Loki-kubernetes",
+        "type": "loki",
+        "typeName": "Loki",
+        "typeLogoUrl": "/public/app/plugins/datasource/loki/img/loki_icon.svg",
+        "access": "proxy",
+        "url": "http://loki.loki:3100",
+        "user": "",
+        "database": "",
+        "basicAuth": false,
+        "isDefault": false,
+        "jsonData": {
+            "httpHeaderName1": "admin",
+            "httpHeaderName2": "X-Scope-OrgID",
+            "tlsSkipVerify": true
+        },
+        "readOnly": false
+    }
+    # In this case, there is a single Loki datasourceUID
+    # with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee"
+
+
 Configuration
 ^^^^^^^^^^^^^
 
-.. md-tab-set::
 
-  .. md-tab-item:: Robusta Helm Chart
+.. md-tab-set::
 
-    Add the following to **Robusta's Helm values:**
+  .. md-tab-item:: Robusta Helm Chat
 
     .. code-block:: yaml
 
-        holmes:
-          toolsets:
-            grafana/loki:
-              enabled: true
-              config:
-                api_key: <your grafana service account token>
-                url: <your grafana url> # e.g. https://acme-corp.grafana.net 
-
-    .. include:: ./_toolset_configuration.inc.rst
+      holmes:
+        toolsets:
+          grafana/loki:
+            enabled: true
+            config:
+              api_key: <your grafana API key>
+              url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
+              grafana_datasource_uid: <the UID of the loki data source in Grafana>
+              labels:
+                pod: "pod"
+                namespace: "namespace"
 
-    You can optionally tweak the search terms used by the toolset. This is only needed if your Loki logs settings for pod,
-    namespace and node differ from the defaults listed below. To do so, add these search keys to the configuration:
+          kubernetes/logs:
+            enabled: false # Disable HolmesGPT's default logging mechanism
 
-    .. code-block:: yaml
 
-        holmes:
-          toolsets:
-            grafana/loki:
-              enabled: true
-              config:
-                api_key: <your grafana service account token>
-                url: <your grafana url> # e.g. https://acme-corp.grafana.net 
-                pod_name_search_key: "pod"
-                namespace_search_key: "namespace"
-                node_name_search_key: "node"
+    .. include:: ./_toolset_configuration.inc.rst
 
   .. md-tab-item:: Holmes CLI
 
@@ -72,29 +109,45 @@ Configuration
         grafana/loki:
           enabled: true
           config:
-              api_key: <your grafana service account token>
-              url: <your grafana url> # e.g. https://acme-corp.grafana.net 
-    
-    You can optionally tweak the search terms used by the toolset. This is only needed if your Loki logs settings for pod,
-    namespace and node differ from the defaults listed below. To do so, add these search keys to the configuration:
+            api_key: <your grafana API key>
+            url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
+            grafana_datasource_uid: <the UID of the loki data source in Grafana>
+            labels:
+                pod: "pod"
+                namespace: "namespace"
 
-    .. code-block:: yaml
+        kubernetes/logs:
+          enabled: false # Disable HolmesGPT's default logging mechanism
 
-      toolsets:
-        grafana/loki:
-          enabled: true
-          config:
-              api_key: <your grafana service account token>
-              url: <your grafana url> # e.g. https://acme-corp.grafana.net 
-            pod_name_search_key: "pod"
-            namespace_search_key: "namespace"
-            node_name_search_key: "node"
 
-    To test, run: 
+**Search labels**
+
+You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if your
+Loki logs settings for ``pod``, and ``namespace`` differ from the defaults in the example above.
+
+Use the following commands to list Loki's labels and determine which ones to use:
+
+.. code-block:: bash
+
+    # Make Loki accessible locally
+    kubectl port-forward svc/loki 3100:3100
+
+    # List all labels. You may have to add the -H 'X-Scope-OrgID:<org id>' option with a valid org id
+    curl http://localhost:3100/loki/api/v1/labels
+
+
+
+If Loki is your primary datasource for logs, it is **advised** to disable the default HolmesGPT logging
+tool by disabling the ``kubernetes/logs`` toolset. Without this. HolmesGPT may still use kubectl to
+fetch logs instead of Loki.
+
+.. code-block:: yaml
+
+    holmes:
+        toolsets:
+            kubernetes/logs:
+                enabled: false
 
-    .. code-block:: yaml
-      
-        holmes ask "Which applications have an issue and check the logs on loki for the reasons"
 
 Capabilities
 ^^^^^^^^^^^^
@@ -107,14 +160,7 @@ Capabilities
 
    * - Tool Name
      - Description
-   * - list_loki_datasources
-     - Fetches the Loki data sources in Grafana
-   * - fetch_loki_logs_by_node
-     - Fetches the Loki logs for a given node
-   * - fetch_loki_logs_by_label
-     - Fetches the Loki logs for a label and value from a Tempo trace
-   * - fetch_loki_logs_by_pod
-     - Fetches the Loki logs for a given pod
-
-
-
+   * - fetch_loki_logs_for_resource
+     - Fetches the Loki logs for a given kubernetes resource
+   * - fetch_loki_logs
+     - Fetches Loki logs from any query
diff --git a/docs/configuration/holmesgpt/toolsets/grafanatempo.rst b/docs/configuration/holmesgpt/toolsets/grafanatempo.rst
@@ -18,12 +18,53 @@ Check out this `video <https://www.loom.com/share/f969ab3af509444693802254ab0407
 Getting Grafana URL
 -----------------------
 
-You can find the Grafana URL required for Loki and Tempo in your Grafana cloud account settings. 
+You can find the Grafana URL required for Tempo in your Grafana cloud account settings.
 
 .. image:: /images/grafana-url-for-holmes.png
   :width: 600
   :align: center
 
+
+Obtaining the datasource UID
+-----------------------
+
+You may have multiple Tempo data sources setup in Grafana. HolmesGPT uses a single Tempo datasource to
+fetch the traces and it needs to know the UID of this datasource.
+
+A simple way to get the datasource UID is to access the Grafana API by running the following request:
+
+.. code-block:: bash
+
+    # port forward if you are using Robusta's grafana from your kubernetes cluster
+    kubectl port-forward svc/robusta-grafana 3000:80
+
+    # List the loki data sources
+    curl -s -u <username>:<password> http://localhost:3000/api/datasources | jq '.[] | select(.type == "loki")'
+    {
+        "id": 2,
+        "uid": "klja8hsa-8a9c-4b35-1230-7baab22b02ee",
+        "orgId": 1,
+        "name": "Tempo-kubernetes",
+        "type": "tempo",
+        "typeName": "Tempo",
+        "typeLogoUrl": "/public/app/plugins/datasource/tempo/img/tempo_icon.svg",
+        "access": "proxy",
+        "url": "http://tempo.tempo:3100",
+        "user": "",
+        "database": "",
+        "basicAuth": false,
+        "isDefault": false,
+        "jsonData": {
+            "httpHeaderName1": "admin",
+            "httpHeaderName2": "X-Scope-OrgID",
+            "tlsSkipVerify": true
+        },
+        "readOnly": false
+    }
+    # In this case, there is a single Tempo datasourceUID
+    # with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee"
+
+
 Configuration
 ^^^^^^^^^^^^^
 
@@ -40,6 +81,7 @@ Configuration
             config:
               api_key: <your grafana API key>
               url: https://grafana-url
+              grafana_datasource_uid: <the UID of the tempo data source in Grafana>
 
     .. include:: ./_toolset_configuration.inc.rst
 
@@ -54,12 +96,13 @@ Configuration
           enabled: true
           config:
               api_key: <your grafana service account token>
-              url: <your grafana url> # e.g. https://acme-corp.grafana.net 
+              url: <your grafana url> # e.g. https://acme-corp.grafana.net
+              grafana_datasource_uid: <the UID of the tempo data source in Grafana>
 
-    To test, run: 
+    To test, run:
 
     .. code-block:: yaml
-      
+
         holmes ask "The payments DB is very slow, check tempo for any trace data"
 
 Capabilities
@@ -78,4 +121,3 @@ Capabilities
      - Lists Tempo traces ids that exceed a specified minimum duration in a given time range
    * - fetch_tempo_trace_by_id
      - Retrieves detailed information about a Tempo trace using its trace ID. Use this to investigate a trace.
-
diff --git a/docs/configuration/holmesgpt/toolsets/kubernetes.rst b/docs/configuration/holmesgpt/toolsets/kubernetes.rst
@@ -52,13 +52,22 @@ Capabilities
    * - kubernetes_jq_query
      - Query Kubernetes resources using jq filters
 
+.. _toolset_kubernetes_logs:
 
 Logs :checkmark:`_`
 -------------------
 
-.. include:: ./_toolset_enabled_by_default.inc.rst
+:checkmark:`_`: **This toolset is enabled by default**. You do not need to configure
+it.
+
+By enabling this toolset, HolmesGPT will be able to read kubernetes pod logs.
+
+You may disable this toolset if you provide Holmes with an alternative toolset to
+fetch logs from your kubernetes cluster and applications. The available built-in
+toolsets that provide logging are:
+
+* :ref:`grafana/loki <toolset_grafana_loki>`: Access Loki logs by proxying through a Grafana instance
 
-Read kubernetes pod logs.
 
 Configuration
 ^^^^^^^^^^^^^
diff --git a/docs/configuration/holmesgpt/toolsets/prometheus.rst b/docs/configuration/holmesgpt/toolsets/prometheus.rst
@@ -1,3 +1,5 @@
+.. _toolset_prometheus:
+
 Prometheus
 ==========
 

Original file line number	Diff line number	Diff line change
`@@ -1,3 +1,5 @@`
	`1`	`+.. _toolset_prometheus:`
	`2`	`+`
`1`	`3`	`Prometheus`
`2`	`4`	`==========`
`3`	`5`