Merge branch 'master' into readme-update-tests

Author: pavangudiwada

.gitignore

@@ -21,3 +21,5 @@ skaffold.dev.yaml
 tests/last_used_tag.txt
 pytest.ini
 gen-config-test
+
+**/.claude/settings.local.json

docs/configuration/alertmanager-integration/_pull_integration.rst

@@ -24,6 +24,12 @@ If Robusta fails to auto-detect the Prometheus and Alertmanager urls - and you s
         # prometheus_auth: ...
         # alertmanager_auth: ...
 
+        # If using a multi-tenant prometheus or alertmanager, pass the org id to all queries
+        # prometheus_additional_headers:
+        #   X-Scope-OrgID: <org id>
+        # alertmanager_additional_headers:
+        #   X-Scope-OrgID: <org id>
+
 .. code-annotations::
     1. Example: http://alertmanager-Helm_release_name-kube-prometheus-alertmanager.default.svc.cluster.local:9093.
     2. Example: http://Helm_Release_Name-kube-prometheus-prometheus.default.svc.cluster.local:9090

docs/configuration/alertmanager-integration/_testing_integration.rst

@@ -3,12 +3,43 @@ Verify it Works
 
 Send a dummy alert to AlertManager:
 
-.. code-block:: bash
+.. tab-set::
 
-    robusta demo-alert
+    .. tab-item:: Robusta CLI
+
+        If you have the Robusta CLI installed, you can send a test alert using the following command:
+
+        .. code-block:: bash
+
+            robusta demo-alert
+
+    .. tab-item:: Robusta UI
+
+        In the Robusta UI, go to the "Clusters" tab, choose the right cluster and click "Simulate Alert".
+        
+        .. image:: /images/robusta-ui-simulate-alert-1.png
+           :alt: Choose the cluster
+           :width: 900
+           :align: center
+
+        Then 
+
+        1. Check **Send alert with no resource**.
+        2. Provide a name for the alert in the **Alert name (identifier)** field (e.g., "Testing Prod AlertManager").
+        3. Select **Alert Manager** under the "Send alert to" section.
+        4. Click the **Simulate Alert** button to send the test alert.
+
+        .. image:: /images/robusta-ui-simulate-alert-2.png
+           :alt: Send Test Alert
+           :width: 600
+           :align: center
 
 If everything is setup properly, this alert will reach Robusta. It will show up in the Robusta UI, Slack, and other configured sinks.
 
+.. note::
+
+    It might take a few minutes for the alert to arrive due to AlertManager's `group_wait` and `group_interval` settings. More info `here <https://prometheus.io/docs/alerting/latest/configuration/#:~:text=How%20long%20to%20wait%20before%20sending%20a%20notification%20about%20new%20alerts%20that%0A%23%20are%20added%20to%20a%20group%20of%20alerts%20for%20which%20an%20initial%20notification%20has%0A%23%20already%20been%20sent>`_.
+
 .. details:: I configured AlertManager, but I'm not receiving alerts?
     :class: warning
 

docs/configuration/alertmanager-integration/coralogix_managed_prometheus.rst

@@ -82,6 +82,12 @@ Metrics querying lets Robusta pull metrics from Coralogix Managed Prometheus.
       # grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (1)
       # alertmanager_flavor: grafana
 
+        # If using a multi-tenant prometheus or alertmanager, pass the org id to all queries
+        # prometheus_additional_headers:
+        #   X-Scope-OrgID: <org id>
+        # alertmanager_additional_headers:
+        #   X-Scope-OrgID: <org id>
+
 .. code-annotations::
     1. This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.
 

docs/configuration/alertmanager-integration/grafana-alert-manager.rst

@@ -83,6 +83,12 @@ Modify and add the following config to ``generated_values.yaml`` and :ref:`updat
         # prometheus_additional_labels:
         #   cluster: 'CLUSTER_NAME_HERE'
 
+        # If using a multi-tenant prometheus or alertmanager, pass the org id to all queries
+        # prometheus_additional_headers:
+        #   X-Scope-OrgID: <org id>
+        # alertmanager_additional_headers:
+        #   X-Scope-OrgID: <org id>
+        
 .. code-annotations::
     1. This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.
 

docs/configuration/alertmanager-integration/victoria-metrics.rst

@@ -8,26 +8,48 @@ You will need to configure two integrations: one to send alerts to Robusta and a
 Send Alerts to Robusta
 ============================
 
-To configure, edit AlertManager's configuration:
+Add the following to your Victoria Metrics Alertmanager configuration (e.g., Helm values file or VMAlertmanagerConfig CRD):
+
+.. code-block:: yaml
+
+    receivers:
+      - name: 'robusta'
+      webhook_configs:
+          - url: 'http://<ROBUSTA-HELM-RELEASE-NAME>-runner.<NAMESPACE>.svc.cluster.local/api/alerts'
+          send_resolved: true # (3)
+
+    route: # (1)
+      routes:
+      - receiver: 'robusta'
+          group_by: [ '...' ]
+          group_wait: 1s
+          group_interval: 1s
+          matchers:
+          - severity =~ ".*"
+          repeat_interval: 4h
+          continue: true # (2)
+
+.. code-annotations::
+    1. Put Robusta's route as the first route, to guarantee it receives alerts. If you can't do so, you must guarantee all previous routes has ``continue: true`` set.
+    2. Keep sending alerts to receivers defined after Robusta.
+    3. Important, so Robusta knows when alerts are resolved.
 
-.. include:: ./_alertmanager-config.rst
 
 .. include:: ./_testing_integration.rst
 
-Configure Metric Querying
+Configure Metrics Querying
 ====================================
 
-Metrics querying lets Robusta pull metrics and create silences.
+Robusta can query metrics and create silences using Victoria Metrics. If both are in the same Kubernetes cluster, Robusta can auto-detect the Victoria Metrics service. To verify, go to the "Apps" tab in Robusta, select an application, and check for usage graphs.
 
-Add the following to ``generated_values.yaml`` and :ref:`update Robusta <Simple Upgrade>`.
+If auto-detection fails you must add the ``prometheus_url`` parameter and :ref:`update Robusta <Simple Upgrade>`.
 
 .. code-block:: yaml
 
     globalConfig: # this line should already exist
         # add the lines below
-        alertmanager_url: "http://ALERT_MANAGER_SERVICE_NAME.NAMESPACE.svc.cluster.local:9093"
-        grafana_url: ""
-        prometheus_url: "http://VICTORIA_METRICS_SERVICE_NAME.NAMESPACE.svc.cluster.local:8429"
+        alertmanager_url: "http://<VM_ALERT_MANAGER_SERVICE_NAME>.<NAMESPACE>.svc.cluster.local:9093" # Example:"http://vmalertmanager-victoria-metrics-vm.default.svc.cluster.local:9093/"        
+        prometheus_url: "http://VM_Metrics_SERVICE_NAME.NAMESPACE.svc.cluster.local:8429" # Example:"http://vmsingle-vmks-victoria-metrics-k8s-stack.default.svc.cluster.local:8429"
         # Add any labels that are relevant to the specific cluster (optional)
         # prometheus_additional_labels:
         #   cluster: 'CLUSTER_NAME_HERE'
@@ -39,6 +61,12 @@ Add the following to ``generated_values.yaml`` and :ref:`update Robusta <Simple
         # grafana_api_key: <YOUR GRAFANA EDITOR API KEY> # (1)
         # alertmanager_flavor: grafana
 
+        # If using a multi-tenant prometheus or alertmanager, pass the org id to all queries
+        # prometheus_additional_headers:
+        #   X-Scope-OrgID: <org id>
+        # alertmanager_additional_headers:
+        #   X-Scope-OrgID: <org id>
+
 .. code-annotations::
     1. This is necessary for Robusta to create silences when using Grafana Alerts, because of minor API differences in the AlertManager embedded in Grafana.
 

docs/configuration/holmesgpt/toolsets/aws.rst

@@ -74,21 +74,7 @@ Configuration
 
   .. md-tab-item:: Robusta Helm Chart
 
-    .. code-block:: yaml
-
-        holmes:
-            additionalEnvVars:
-                - name: AWS_ACCESS_KEY_ID
-                  value: AKIXDDDSDSdSA
-                - name: AWS_SECRET_ACCESS_KEY
-                  value: =wJalrXUtnFEMI/KNG/bPxRfiCYEXAMPLEKEY
-                - name: AWS_DEFAULT_REGION
-                  value: us-west-2
-            toolsets:
-                aws/rds:
-                    enabled: true
-
-    .. include:: ./_toolset_configuration.inc.rst
+    This builtin toolset is currently only available in HolmesGPT CLI.
 
   .. md-tab-item:: Holmes CLI
 

docs/configuration/holmesgpt/toolsets/coralogix_logs.rst

@@ -29,7 +29,7 @@ Configuration
 
 .. md-tab-set::
 
-  .. md-tab-item:: Robusta Helm Chat
+  .. md-tab-item:: Robusta Helm Chart
 
     .. code-block:: yaml
 
@@ -68,7 +68,8 @@ Configuration
 Advanced Configuration
 ^^^^^^^^^^^^^^^^^^^^^^
 
-**Frequent logs and archive**
+Frequent logs and archive
+****************************
 
 By default, holmes fetched the logs from the `Frequent search <https://coralogix.com/docs/user-guides/account-management/tco-optimizer/logs/#frequent-search-data-high-priority>`_ 
 tier and only fetch logs from the `Archive` tier if the frequent search returned no result.
@@ -98,7 +99,8 @@ Here is a description of each possible log retrieval methodology:
 - **FREQUENT_SEARCH_FALLBACK** Search logs in the archive first. If there are no results, fallback to searching the frequent logs.
 - **BOTH_FREQUENT_SEARCH_AND_ARCHIVE** Always use both the frequent search and the archive to fetch logs. The result contains merged data which is deduplicated and sorted by timestamp.
 
-**Search labels**
+Search labels
+***************
 
 You can tweak the labels used by the toolset to identify kubernetes resources. This is **optional** and only needed if your
 logs settings for ``pod``, ``namespace``, ``application`` and ``subsystem`` differ from the defaults in the example below.
@@ -124,7 +126,8 @@ You can verify what labels to use by attempting to run a query in the coralogix
   :align: center
 
 
-**Disabling the default toolset**
+Disabling the default toolset
+*********************************
 
 If Coralogix is your primary datasource for logs, it is **advised** to disable the default HolmesGPT logging
 tool by disabling the ``kubernetes/logs`` toolset. Without this. HolmesGPT may still use kubectl to