Skip to content

Commit 859a48c

Browse files
nhermentnherment
authored
doc: update holmes docs for Loki and Tempo (#1808)
1 parent dfc97f6 commit 859a48c

2 files changed

Lines changed: 208 additions & 35 deletions

File tree

‎docs/configuration/holmesgpt/toolsets/grafanaloki.rst‎

Lines changed: 101 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,10 @@
11
.. _toolset_grafana_loki:
22

3-
Loki (Grafana)
4-
===============
3+
Loki
4+
====
55

6-
By enabling this toolset, HolmesGPT will fetch node and pods logs from `Loki <https://grafana.com/oss/loki/>`_
7-
by proxying through a `Grafana <https://grafana.com/oss/grafana/>`_ instance.
6+
By enabling this toolset, HolmesGPT will fetch node and pods logs from `Loki <https://grafana.com/oss/loki/>`_.
7+
Loki can be accessed directly or by proxying through a `Grafana <https://grafana.com/oss/grafana/>`_ instance.
88

99
You **should** enable this toolset to replace the default :ref:`kubernetes/logs <toolset_kubernetes_logs>`
1010
toolset if all your kubernetes/pod logs are consolidated inside Loki. It will make it easier for HolmesGPT
@@ -13,8 +13,14 @@ to fetch incident logs, including the ability to precisely consult past logs.
1313

1414
.. include:: ./_toolsets_that_provide_logging.inc.rst
1515

16+
17+
Proxying through Grafana
18+
^^^^^^^^^^^^^^^^^^^^^^^^
19+
20+
This is the recommended approach because we intend to add more capabilities to the toolset that are only available with Grafana.
21+
1622
Prerequisites
17-
^^^^^^^^^^^^^
23+
-------------
1824

1925
A `Grafana service account token <https://grafana.com/docs/grafana/latest/administration/service-accounts/>`_
2026
with the following permissions:
@@ -24,8 +30,7 @@ with the following permissions:
2430

2531
Check out this `video <https://www.loom.com/share/f969ab3af509444693802254ab040791?sid=aa8b3c65-2696-4f69-ae47-bb96e8e03c47>`_ on creating a Grafana service account token.
2632

27-
Getting Grafana URL
28-
-----------------------
33+
**Getting Grafana URL**
2934

3035
You can find the Grafana URL required for Loki in your Grafana cloud account settings.
3136

@@ -34,8 +39,7 @@ You can find the Grafana URL required for Loki in your Grafana cloud account set
3439
:align: center
3540

3641

37-
Obtaining the datasource UID
38-
-----------------------------------
42+
**Obtaining the datasource UID**
3943

4044
You may have multiple Loki data sources setup in Grafana. HolmesGPT uses a single Loki datasource to
4145
fetch the logs and it needs to know the UID of this datasource.
@@ -74,8 +78,8 @@ A simple way to get the datasource UID is to access the Grafana API by running t
7478
# with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee"
7579
7680
77-
Configuration
78-
^^^^^^^^^^^^^
81+
Configuration (grafana proxy)
82+
-----------------------------
7983

8084

8185
.. md-tab-set::
@@ -92,9 +96,6 @@ Configuration
9296
api_key: <your grafana API key>
9397
url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
9498
grafana_datasource_uid: <the UID of the loki data source in Grafana>
95-
labels:
96-
pod: "pod"
97-
namespace: "namespace"
9899
99100
kubernetes/logs:
100101
enabled: false # Disable HolmesGPT's default logging mechanism
@@ -115,19 +116,102 @@ Configuration
115116
api_key: <your grafana API key>
116117
url: https://xxxxxxx.grafana.net # Your Grafana cloud account URL
117118
grafana_datasource_uid: <the UID of the loki data source in Grafana>
118-
labels:
119-
pod: "pod"
120-
namespace: "namespace"
121119
122120
kubernetes/logs:
123121
enabled: false # Disable HolmesGPT's default logging mechanism
124122
123+
Direct connection
124+
^^^^^^^^^^^^^^^^^
125+
126+
The toolset can directly connect to a Loki instance without proxying through a Grafana instance.
127+
This is done by not setting the ``grafana_datasource_uid`` field. Not setting this field makes HolmesGPT
128+
assume that it is directly connecting to Loki.
129+
130+
Configuration (direct connection)
131+
---------------------------------
132+
133+
.. md-tab-set::
134+
135+
.. md-tab-item:: Robusta Helm Chat
136+
137+
.. code-block:: yaml
138+
139+
holmes:
140+
toolsets:
141+
grafana/loki:
142+
enabled: true
143+
config:
144+
url: http://loki.logging
145+
headers:
146+
X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled
147+
148+
kubernetes/logs:
149+
enabled: false # Disable HolmesGPT's default logging mechanism
150+
151+
152+
.. include:: ./_toolset_configuration.inc.rst
153+
154+
.. md-tab-item:: Holmes CLI
155+
156+
Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist:
157+
158+
.. code-block:: yaml
159+
160+
toolsets:
161+
grafana/loki:
162+
enabled: true
163+
config:
164+
url: http://loki.logging
165+
headers:
166+
X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if loki multitenancy is enabled
167+
168+
kubernetes/logs:
169+
enabled: false # Disable HolmesGPT's default logging mechanism
170+
171+
172+
Advanced configuration
173+
^^^^^^^^^^^^^^^^^^^^^^
125174

126175
**Search labels**
127176

128177
You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if your
129178
Loki logs settings for ``pod``, and ``namespace`` differ from the defaults in the example above.
130179

180+
181+
.. md-tab-set::
182+
183+
.. md-tab-item:: Robusta Helm Chat
184+
185+
.. code-block:: yaml
186+
187+
holmes:
188+
toolsets:
189+
grafana/loki:
190+
enabled: true
191+
config:
192+
url: ...
193+
labels:
194+
pod: "pod"
195+
namespace: "namespace"
196+
197+
.. include:: ./_toolset_configuration.inc.rst
198+
199+
.. md-tab-item:: Holmes CLI
200+
201+
Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist:
202+
203+
.. code-block:: yaml
204+
205+
toolsets:
206+
grafana/loki:
207+
enabled: true
208+
config:
209+
url: ...
210+
labels:
211+
pod: "pod"
212+
namespace: "namespace"
213+
214+
131215
Use the following commands to list Loki's labels and determine which ones to use:
132216

133217
.. code-block:: bash

‎docs/configuration/holmesgpt/toolsets/grafanatempo.rst‎

Lines changed: 107 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,18 @@
11

2-
Tempo (Grafana)
3-
================
2+
Tempo
3+
=====
44

55
By enabling this toolset, HolmesGPT will be able to fetch trace information from Grafana
66
Tempo to debug performance related issues, like high latency in your application.
77

8+
Proxying through Grafana
9+
^^^^^^^^^^^^^^^^^^^^^^^^
10+
11+
This is the recommended approach because we intend to add more capabilities to the toolset that are only available with Grafana.
12+
813
Prerequisites
9-
^^^^^^^^^^^^^
14+
-------------
15+
1016
A `Grafana service account token <https://grafana.com/docs/grafana/latest/administration/service-accounts/>`_
1117
with the following permissions:
1218

@@ -15,17 +21,15 @@ with the following permissions:
1521

1622
Check out this `video <https://www.loom.com/share/f969ab3af509444693802254ab040791?sid=aa8b3c65-2696-4f69-ae47-bb96e8e03c47>`_ on creating a Grafana service account token.
1723

18-
Getting Grafana URL
19-
-----------------------
24+
**Getting Grafana URL**
2025

2126
You can find the Grafana URL required for Tempo in your Grafana cloud account settings.
2227

2328
.. image:: /images/grafana-url-for-holmes.png
2429
:width: 600
2530
:align: center
2631

27-
Obtaining the datasource UID
28-
--------------------------------
32+
**Obtaining the datasource UID**
2933

3034
You may have multiple Tempo data sources setup in Grafana. HolmesGPT uses a single Tempo datasource to
3135
fetch the traces and it needs to know the UID of this datasource.
@@ -60,8 +64,8 @@ A simple way to get the datasource UID is to access the Grafana API by running t
6064
# In this case, there is a single Tempo datasourceUID
6165
# with UID "klja8hsa-8a9c-4b35-1230-7baab22b02ee"
6266
63-
Configuration
64-
^^^^^^^^^^^^^
67+
Configuration (grafana proxy)
68+
-----------------------------
6569

6670
.. md-tab-set::
6771

@@ -96,28 +100,113 @@ Configuration
96100
grafana/tempo:
97101
enabled: true
98102
config:
99-
api_key: <your grafana service account token>
100-
url: <your grafana url> # e.g. https://acme-corp.grafana.net
101-
grafana_datasource_uid: <the UID of the tempo data source in Grafana>
102-
labels:
103-
pod: "k8s.pod.name"
104-
namespace: "k8s.namespace.name"
105-
deployment: "k8s.deployment.name"
106-
node: "k8s.node.name"
107-
service: "service.name"
103+
api_key: <your grafana service account token>
104+
url: <your grafana url> # e.g. https://acme-corp.grafana.net
105+
grafana_datasource_uid: <the UID of the tempo data source in Grafana>
108106
109107
To test, run:
110108

111109
.. code-block:: yaml
112110
113111
holmes ask "The payments DB is very slow, check tempo for any trace data"
114112
113+
Direct connection
114+
^^^^^^^^^^^^^^^^^
115+
116+
The toolset can directly connect to a Tempo instance without proxying through a Grafana instance.
117+
This is done by not setting the ``grafana_datasource_uid`` field. Not setting this field makes HolmesGPT
118+
assume that it is directly connecting to Tempo.
119+
120+
Configuration (direct connection)
121+
---------------------------------
122+
123+
.. md-tab-set::
124+
125+
.. md-tab-item:: Robusta Helm Chat
126+
127+
.. code-block:: yaml
128+
129+
holmes:
130+
toolsets:
131+
grafana/tempo:
132+
enabled: true
133+
config:
134+
url: http://tempo.monitoring
135+
headers:
136+
X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if tempo multitenancy is enabled
137+
138+
kubernetes/logs:
139+
enabled: false # Disable HolmesGPT's default logging mechanism
140+
141+
142+
.. include:: ./_toolset_configuration.inc.rst
143+
144+
.. md-tab-item:: Holmes CLI
145+
146+
Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist:
147+
148+
.. code-block:: yaml
149+
150+
toolsets:
151+
grafana/tempo:
152+
enabled: true
153+
config:
154+
url: http://tempo.monitoring
155+
headers:
156+
X-Scope-OrgID: "<tenant id>" # Set the X-Scope-OrgID if tempo multitenancy is enabled
157+
158+
kubernetes/logs:
159+
enabled: false # Disable HolmesGPT's default logging mechanism
160+
161+
162+
Advanced configuration
163+
^^^^^^^^^^^^^^^^^^^^^^
164+
115165
**Search labels**
116166

117167
You can tweak the labels used by the toolset to identify kubernetes resources. This is only needed if the trace
118168
labels differ from the defaults in the example above.
119169

120170

171+
.. md-tab-set::
172+
173+
.. md-tab-item:: Robusta Helm Chat
174+
175+
.. code-block:: yaml
176+
177+
holmes:
178+
toolsets:
179+
grafana/tempo:
180+
enabled: true
181+
config:
182+
url: ...
183+
labels:
184+
pod: "k8s.pod.name"
185+
namespace: "k8s.namespace.name"
186+
deployment: "k8s.deployment.name"
187+
node: "k8s.node.name"
188+
service: "service.name"
189+
190+
.. include:: ./_toolset_configuration.inc.rst
191+
192+
.. md-tab-item:: Holmes CLI
193+
194+
Add the following to **~/.holmes/config.yaml**, creating the file if it doesn't exist:
195+
196+
.. code-block:: yaml
197+
198+
toolsets:
199+
grafana/tempo:
200+
enabled: true
201+
config:
202+
url: ...
203+
labels:
204+
pod: "k8s.pod.name"
205+
namespace: "k8s.namespace.name"
206+
deployment: "k8s.deployment.name"
207+
node: "k8s.node.name"
208+
service: "service.name"
209+
121210
Capabilities
122211
^^^^^^^^^^^^
123212
.. include:: ./_toolset_capabilities.inc.rst

0 commit comments

Comments
 (0)