diff --git a/docs/configuration/holmesgpt/toolsets/grafanaloki.rst b/docs/configuration/holmesgpt/toolsets/grafanaloki.rst index 730373936..9d0910a32 100644 --- a/docs/configuration/holmesgpt/toolsets/grafanaloki.rst +++ b/docs/configuration/holmesgpt/toolsets/grafanaloki.rst @@ -1,10 +1,10 @@ .. _toolset_grafana_loki: -Loki (Grafana) -=============== +Loki +==== -By enabling this toolset, HolmesGPT will fetch node and pods logs from `Loki `_ -by proxying through a `Grafana `_ instance. +By enabling this toolset, HolmesGPT will fetch node and pods logs from `Loki `_. +Loki can be accessed directly or by proxying through a `Grafana `_ instance. You **should** enable this toolset to replace the default :ref:`kubernetes/logs ` toolset if all your kubernetes/pod logs are consolidated inside Loki. It will make it easier for HolmesGPT @@ -13,8 +13,14 @@ to fetch incident logs, including the ability to precisely consult past logs. .. include:: ./_toolsets_that_provide_logging.inc.rst + +Proxying through Grafana +^^^^^^^^^^^^^^^^^^^^^^^^ + +This is the recommended approach because we intend to add more capabilities to the toolset that are only available with Grafana. + Prerequisites -^^^^^^^^^^^^^ +------------- A `Grafana service account token `_ with the following permissions: @@ -24,8 +30,7 @@ with the following permissions: Check out this `video `_ on creating a Grafana service account token. -Getting Grafana URL ------------------------ +**Getting Grafana URL** You can find the Grafana URL required for Loki in your Grafana cloud account settings. @@ -34,8 +39,7 @@ You can find the Grafana URL required for Loki in your Grafana cloud account set :align: center -Obtaining the datasource UID ------------------------------------ +**Obtaining the datasource UID** You may have multiple Loki data sources setup in Grafana. HolmesGPT uses a single Loki datasource to fetch the logs and it needs to know the UID of this datasource. @@ -74,8 +78,8 @@ A simple way to get the datasource UID is to access the Grafana API by running t # with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee" -Configuration -^^^^^^^^^^^^^ +Configuration (grafana proxy) +----------------------------- .. md-tab-set:: @@ -92,9 +96,6 @@ Configuration api_key: url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL grafana_datasource_uid: - labels: - pod: "pod" - namespace: "namespace" kubernetes/logs: enabled: false # Disable HolmesGPT's default logging mechanism @@ -115,19 +116,102 @@ Configuration api_key: url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL grafana_datasource_uid: - labels: - pod: "pod" - namespace: "namespace" kubernetes/logs: enabled: false # Disable HolmesGPT's default logging mechanism +Direct connection +^^^^^^^^^^^^^^^^^ + +The toolset can directly connect to a Loki instance without proxying through a Grafana instance. +This is done by not setting the ``grafana_datasource_uid`` field. Not setting this field makes HolmesGPT +assume that it is directly connecting to Loki. + +Configuration (direct connection) +--------------------------------- + +.. md-tab-set:: + + .. md-tab-item:: Robusta Helm Chat + + .. code-block:: yaml + + holmes: + toolsets: + grafana/loki: + enabled: true + config: + url: http://loki.logging + headers: + X-Scope-OrgID: "" # Set the X-Scope-OrgID if loki multitenancy is enabled + + kubernetes/logs: + enabled: false # Disable HolmesGPT's default logging mechanism + + + .. include:: ./_toolset_configuration.inc.rst + + .. md-tab-item:: Holmes CLI + + Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist: + + .. code-block:: yaml + + toolsets: + grafana/loki: + enabled: true + config: + url: http://loki.logging + headers: + X-Scope-OrgID: "" # Set the X-Scope-OrgID if loki multitenancy is enabled + + kubernetes/logs: + enabled: false # Disable HolmesGPT's default logging mechanism + + +Advanced configuration +^^^^^^^^^^^^^^^^^^^^^^ **Search labels** You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if your Loki logs settings for ``pod``, and ``namespace`` differ from the defaults in the example above. + +.. md-tab-set:: + + .. md-tab-item:: Robusta Helm Chat + + .. code-block:: yaml + + holmes: + toolsets: + grafana/loki: + enabled: true + config: + url: ... + labels: + pod: "pod" + namespace: "namespace" + + .. include:: ./_toolset_configuration.inc.rst + + .. md-tab-item:: Holmes CLI + + Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist: + + .. code-block:: yaml + + toolsets: + grafana/loki: + enabled: true + config: + url: ... + labels: + pod: "pod" + namespace: "namespace" + + Use the following commands to list Loki's labels and determine which ones to use: .. code-block:: bash diff --git a/docs/configuration/holmesgpt/toolsets/grafanatempo.rst b/docs/configuration/holmesgpt/toolsets/grafanatempo.rst index 928c18ff7..7903b4c9d 100644 --- a/docs/configuration/holmesgpt/toolsets/grafanatempo.rst +++ b/docs/configuration/holmesgpt/toolsets/grafanatempo.rst @@ -1,12 +1,18 @@ -Tempo (Grafana) -================ +Tempo +===== By enabling this toolset, HolmesGPT will be able to fetch trace information from Grafana Tempo to debug performance related issues, like high latency in your application. +Proxying through Grafana +^^^^^^^^^^^^^^^^^^^^^^^^ + +This is the recommended approach because we intend to add more capabilities to the toolset that are only available with Grafana. + Prerequisites -^^^^^^^^^^^^^ +------------- + A `Grafana service account token `_ with the following permissions: @@ -15,8 +21,7 @@ with the following permissions: Check out this `video `_ on creating a Grafana service account token. -Getting Grafana URL ------------------------ +**Getting Grafana URL** You can find the Grafana URL required for Tempo in your Grafana cloud account settings. @@ -24,8 +29,7 @@ You can find the Grafana URL required for Tempo in your Grafana cloud account se :width: 600 :align: center -Obtaining the datasource UID --------------------------------- +**Obtaining the datasource UID** You may have multiple Tempo data sources setup in Grafana. HolmesGPT uses a single Tempo datasource to fetch the traces and it needs to know the UID of this datasource. @@ -60,8 +64,8 @@ A simple way to get the datasource UID is to access the Grafana API by running t # In this case, there is a single Tempo datasourceUID # with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee" -Configuration -^^^^^^^^^^^^^ +Configuration (grafana proxy) +----------------------------- .. md-tab-set:: @@ -96,15 +100,9 @@ Configuration grafana/tempo: enabled: true config: - api_key: - url: # e.g. https://acme-corp.grafana.net - grafana_datasource_uid: - labels: - pod: "k8s.pod.name" - namespace: "k8s.namespace.name" - deployment: "k8s.deployment.name" - node: "k8s.node.name" - service: "service.name" + api_key: + url: # e.g. https://acme-corp.grafana.net + grafana_datasource_uid: To test, run: @@ -112,12 +110,103 @@ Configuration holmes ask "The payments DB is very slow, check tempo for any trace data" +Direct connection +^^^^^^^^^^^^^^^^^ + +The toolset can directly connect to a Tempo instance without proxying through a Grafana instance. +This is done by not setting the ``grafana_datasource_uid`` field. Not setting this field makes HolmesGPT +assume that it is directly connecting to Tempo. + +Configuration (direct connection) +--------------------------------- + +.. md-tab-set:: + + .. md-tab-item:: Robusta Helm Chat + + .. code-block:: yaml + + holmes: + toolsets: + grafana/tempo: + enabled: true + config: + url: http://tempo.monitoring + headers: + X-Scope-OrgID: "" # Set the X-Scope-OrgID if tempo multitenancy is enabled + + kubernetes/logs: + enabled: false # Disable HolmesGPT's default logging mechanism + + + .. include:: ./_toolset_configuration.inc.rst + + .. md-tab-item:: Holmes CLI + + Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist: + + .. code-block:: yaml + + toolsets: + grafana/tempo: + enabled: true + config: + url: http://tempo.monitoring + headers: + X-Scope-OrgID: "" # Set the X-Scope-OrgID if tempo multitenancy is enabled + + kubernetes/logs: + enabled: false # Disable HolmesGPT's default logging mechanism + + +Advanced configuration +^^^^^^^^^^^^^^^^^^^^^^ + **Search labels** You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if the trace labels differ from the defaults in the example above. +.. md-tab-set:: + + .. md-tab-item:: Robusta Helm Chat + + .. code-block:: yaml + + holmes: + toolsets: + grafana/tempo: + enabled: true + config: + url: ... + labels: + pod: "k8s.pod.name" + namespace: "k8s.namespace.name" + deployment: "k8s.deployment.name" + node: "k8s.node.name" + service: "service.name" + + .. include:: ./_toolset_configuration.inc.rst + + .. md-tab-item:: Holmes CLI + + Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist: + + .. code-block:: yaml + + toolsets: + grafana/tempo: + enabled: true + config: + url: ... + labels: + pod: "k8s.pod.name" + namespace: "k8s.namespace.name" + deployment: "k8s.deployment.name" + node: "k8s.node.name" + service: "service.name" + Capabilities ^^^^^^^^^^^^ .. include:: ./_toolset_capabilities.inc.rst