Skip to content

Commit 544734a

Browse files
Cortex-DMSCortex-DMS
authored
Feature/observability upgrade components (#273)
Updated Observability component versions in install guides and importing and configuring additional default dashboard, added FAQ for changing Logging Level, and upgrade guide
1 parent 08dae99 commit 544734a

84 files changed

Lines changed: 1798 additions & 352 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎content/en/docs/2025.3/FAQs/change-logging-levels/_index.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
---
2+
title: "How do I change the Logging Level?"
3+
linkTitle: "How do I change the Logging Level?"
4+
description: "Instructions on how to change the {{% ctx %}} Logging Level."
5+
weight: 1000
6+
---
7+
8+
{{% ctx %}} Logging Levels are set to `Error` by default for most services. However, levels can be changed whilst {{% ctx %}} is running by sending REST calls to the {{% ctx %}} API Gateway service.
9+
10+
{{% alert title="Note" %}}
11+
Although the Logging Level for the {{% ctx %}} API Gateway service can be increased, it will still always log all API calls regardless of Logging Level configured.
12+
{{% / alert %}}
13+
14+
If increasing the Logging Level, particularly if increasing for everything or for Block Logging, we recommend that this is done only when required. Increasing log levels will lead to an increase in disk usage and disk writes, which can impact the performance of the {{% ctx %}} platform if not monitored.
15+
16+
Whilst increased Logging Levels are in place, we would recommend closely monitoring system performance and disk space usage.
17+
18+
All REST calls to update Logging Levels use Basic Auth and can be made using REST clients such as [Postman][] or using scripts such as PowerShell.
19+
20+
[Postman]: {{< url path="Postman.Downloads.MainDoc" >}}

‎content/en/docs/2025.3/FAQs/change-logging-levels/change-all-logging.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
title: "Change All Logging Levels"
3+
linkTitle: "Change All Logging Levels"
4+
description: "Instructions on how to change the Logging Level for All Logging."
5+
weight: 10
6+
---
7+
8+
# {{% param title %}}
9+
10+
To change the Logging Level for all services, the below PowerShell script can be used to make a REST call against your {{% ctx %}} platform.
11+
12+
1. Run Windows PowerShell ISE as Administrator.
13+
1. Copy the following script into the PowerShell window:
14+
15+
``` powershell
16+
$serverFQDN = "server.domain.com"
17+
$APIGatewayPort = 8722
18+
$loglevel = 4
19+
$user = "BasicAuthUser"
20+
21+
$securePass = Read-Host -Prompt "Enter password for $user" -AsSecureString
22+
23+
$ptr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($securePass)
24+
try {
25+
$plainPass = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($ptr)
26+
27+
$authBytes = [System.Text.Encoding]::ASCII.GetBytes("$user`:$plainPass")
28+
$base64AuthInfo = [Convert]::ToBase64String($authBytes)
29+
}
30+
finally {
31+
[Runtime.InteropServices.Marshal]::ZeroFreeBSTR($ptr)
32+
33+
$plainPass = $null
34+
Remove-Variable -Name plainPass -Force
35+
}
36+
37+
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
38+
$headers.Add("Content-Type", "application/json")
39+
$headers.Add("Accept", "application/json")
40+
$headers.Add("Authorization", "Basic $base64AuthInfo")
41+
$path = "applications/logging"
42+
$body = @"
43+
$loglevel
44+
"@
45+
46+
$response = Invoke-RestMethod "https://${serverFQDN}:$APIGatewayPort/api/v1/default/default/$path" -Method PUT -Headers $headers -Body $body
47+
$response
48+
```
49+
50+
1. Configure the following variables:
51+
* `$serverFQDN` – The fully qualified domain name for the Application Server or Load Balancer
52+
* `$APIGatewayPort` – {{% ctx %}} API Gateway Service Port (8722) or Load Balancer port (typically 443 or 8722)
53+
* `$loglevel` – Desired log level as an integer, `1` is enabled and `4` is the default error level
54+
* `$user` – {{% ctx %}} API Gateway Basic Auth Username
55+
56+
1. Execute the script, entering the Basic Auth User's password when prompted.
57+
1. Confirm success response:
58+
59+
If the call was successful, there should be no errors and the following response should be received
60+
61+
``` powershell
62+
LogLevel was successfully configured.
63+
```

‎content/en/docs/2025.3/FAQs/change-logging-levels/change-block-logging.md‎

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
title: "Change Block Logging Level"
3+
linkTitle: "Change Block Logging Level"
4+
description: "Instructions on how to change the Logging Level for Block Logging."
5+
weight: 5
6+
---
7+
8+
# {{% param title %}}
9+
10+
Block Logging in {{% ctx %}} logs the following Block Properties by default regardless of the log level configured:
11+
12+
* Execute Data Command
13+
* Command > commandText
14+
* Command > parameters
15+
* Execute Http Request
16+
* Http Request
17+
* Http Response
18+
* Execute Soap Request
19+
* Soap Request
20+
* Soap Response
21+
* Execute PowerShell Script
22+
* Script
23+
* Parameters
24+
* Outputs
25+
* Records
26+
* Execute Ssh Command
27+
* Command
28+
* Response
29+
* Ssh Logs
30+
31+
To change the Logging Level for Block Logging so that every block is logged when executed, the below PowerShell script can be used to make a REST call against your {{% ctx %}} platform.
32+
33+
1. Run Windows PowerShell ISE as Administrator.
34+
1. Copy the following script into the PowerShell window:
35+
36+
``` powershell
37+
$serverFQDN = "server.domain.com"
38+
$APIGatewayPort = 8722
39+
$loglevel = 4
40+
$user = "BasicAuthUser"
41+
42+
$securePass = Read-Host -Prompt "Enter password for $user" -AsSecureString
43+
44+
$ptr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($securePass)
45+
try {
46+
$plainPass = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($ptr)
47+
48+
$authBytes = [System.Text.Encoding]::ASCII.GetBytes("$user`:$plainPass")
49+
$base64AuthInfo = [Convert]::ToBase64String($authBytes)
50+
}
51+
finally {
52+
[Runtime.InteropServices.Marshal]::ZeroFreeBSTR($ptr)
53+
54+
$plainPass = $null
55+
Remove-Variable -Name plainPass -Force
56+
}
57+
58+
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
59+
$headers.Add("Content-Type", "application/json")
60+
$headers.Add("Accept", "application/json")
61+
$headers.Add("Authorization", "Basic $base64AuthInfo")
62+
$path = "applications/execution/services/engine/blocks/packages/versions/executions/flows/workspaces/blocks/logging"
63+
$body = @"
64+
$loglevel
65+
"@
66+
67+
$response = Invoke-RestMethod "https://${serverFQDN}:$APIGatewayPort/api/v1/default/default/$path" -Method PUT -Headers $headers -Body $body
68+
$response
69+
```
70+
71+
1. Configure the following variables:
72+
* `$serverFQDN` – The fully qualified domain name for the Application Server or Load Balancer
73+
* `$APIGatewayPort` – {{% ctx %}} API Gateway Service Port (8722) or Load Balancer port (typically 443 or 8722)
74+
* `$loglevel` – Desired log level as an integer, `1` is enabled and `4` is the default error level
75+
* `$user` – {{% ctx %}} API Gateway Basic Auth Username
76+
77+
1. Execute the script, entering the Basic Auth User's password when prompted.
78+
1. Confirm success response:
79+
80+
If the call was successful, there should be no errors and the following response should be received
81+
82+
``` powershell
83+
LogLevel was successfully configured.
84+
```

‎content/en/docs/2025.3/FAQs/change-logging-levels/change-flow-logging.md‎

Lines changed: 63 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,63 @@
1+
---
2+
title: "Change Flow Logging Level"
3+
linkTitle: "Change Flow Logging Level"
4+
description: "Instructions on how to change the Logging Level for Flow Logging."
5+
weight: 1
6+
---
7+
8+
# {{% param title %}}
9+
10+
To change the Logging Level for Flow Logging, the below PowerShell script can be used to make a REST call against your {{% ctx %}} platform.
11+
12+
1. Run Windows PowerShell ISE as Administrator.
13+
1. Copy the following script into the PowerShell window:
14+
15+
``` powershell
16+
$serverFQDN = "server.domain.com"
17+
$APIGatewayPort = 8722
18+
$loglevel = 4
19+
$user = "BasicAuthUser"
20+
21+
$securePass = Read-Host -Prompt "Enter password for $user" -AsSecureString
22+
23+
$ptr = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($securePass)
24+
try {
25+
$plainPass = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($ptr)
26+
27+
$authBytes = [System.Text.Encoding]::ASCII.GetBytes("$user`:$plainPass")
28+
$base64AuthInfo = [Convert]::ToBase64String($authBytes)
29+
}
30+
finally {
31+
[Runtime.InteropServices.Marshal]::ZeroFreeBSTR($ptr)
32+
33+
$plainPass = $null
34+
Remove-Variable -Name plainPass -Force
35+
}
36+
37+
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
38+
$headers.Add("Content-Type", "application/json")
39+
$headers.Add("Accept", "application/json")
40+
$headers.Add("Authorization", "Basic $base64AuthInfo")
41+
$path = "applications/execution/services/engine/blocks/packages/versions/executions/flows/logging"
42+
$body = @"
43+
$loglevel
44+
"@
45+
46+
$response = Invoke-RestMethod "https://${serverFQDN}:$APIGatewayPort/api/v1/default/default/$path" -Method PUT -Headers $headers -Body $body
47+
$response
48+
```
49+
50+
1. Configure the following variables:
51+
* `$serverFQDN` – The fully qualified domain name for the Application Server or Load Balancer
52+
* `$APIGatewayPort` – {{% ctx %}} API Gateway Service Port (8722) or Load Balancer port (typically 443 or 8722)
53+
* `$loglevel` – Desired log level as an integer, `1` is enabled and `4` is the default error level
54+
* `$user` – {{% ctx %}} API Gateway Basic Auth Username
55+
56+
1. Execute the script, entering the Basic Auth User's password when prompted.
57+
1. Confirm success response:
58+
59+
If the call was successful, there should be no errors and the following response should be received
60+
61+
``` powershell
62+
LogLevel was successfully configured.
63+
```

‎content/en/docs/2025.3/getting-started/cloud/add-observability-to-innovation/Grafana/architecture.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ weight: 10
1212
| Component | Purpose | Required/Optional |Server Role |
1313
|-----------|---------|----------|------------|
1414
| [Grafana][] | Web application that provides querying and visualisation of data in the form of dashboards. | Required | Grafana&nbsp;Cloud&nbsp;managed&nbsp;service |
15+
| [Grafana Alloy][] | An observability collector that can ingest logs and send them to a Grafana Loki instance. It should be deployed to every machine that has a Microsoft Service Fabric node used by {{% ctx %}}. | Required | Application&nbsp;Server |
1516
| [Grafana&nbsp;Loki][Grafana Loki] | Log aggregation system designed to store and query logs from applications and infrastructure. | Required | Grafana&nbsp;Cloud&nbsp;managed&nbsp;service |
16-
| [Promtail][] | An agent which ships the contents of local logs to a Grafana Loki instance. It should be deployed to every machine that has a Microsoft Service Fabric node used by {{% ctx %}}. | Required | Application&nbsp;Server |
1717

1818
## Recommended Architecture
1919

@@ -27,6 +27,6 @@ The following architecture requires 1..n Application servers and 1 [Grafana Clou
2727

2828
[Prerequisites]: {{< url path="Cortex.GettingStarted.Cloud.AddObservabilityToInnovation.Grafana.Prerequisites.MainDoc" >}}
2929
[Grafana]: {{< url path="Grafana.Products.Grafana.MainDoc" >}}
30+
[Grafana Alloy]: {{< url path="Grafana.Products.Loki.Alloy.MainDoc" >}}
3031
[Grafana Cloud]: {{< url path="Grafana.MainDoc" >}}
3132
[Grafana Loki]: {{< url path="Grafana.Products.Loki.MainDoc" >}}
32-
[Promtail]: {{< url path="Grafana.Products.Loki.Promtail.MainDoc" >}}

‎content/en/docs/2025.3/getting-started/cloud/add-observability-to-innovation/Grafana/import-dashboards.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -32,13 +32,18 @@ Please ensure that the set up for [Grafana][] and [Loki][] have been completed b
3232
1. Click the Dashboards menu item.
3333
1. Click the *New* dropdown and select *Import*.
3434
1. Click on *Upload dashboard JSON file*.
35-
1. Locate the `Flow Execution Requests.json` file extracted from the downloaded `Grafana.Dashboards.zip`.
35+
1. Locate the `Flow Execution Details.json` file extracted from the downloaded `Grafana.Dashboards.zip`.
3636
1. Select the file and click *Open*.
3737
1. Select the folder in Grafana you wish the dashboard to be saved in, e.g. *Cortex*.
3838
1. Select your [configured Loki data source][] from the dropdown menu, e.g. *grafanacloud-cortex-logs*.
3939
1. Click *Import*.
40+
1. Repeat steps 2 - 10 for the `Flow Execution Requests.json` file.
4041
1. Repeat steps 2 - 10 for the `Platform Health.json` file.
4142

43+
{{% alert title="Warning" color="warning" %}}
44+
The *Flow Execution Details* dashboard will only work if `Flow Logging` is enabled on your platform. See {{< ahref path="Cortex.Faqs.ChangeLoggingLevels.FlowLogging.MainDoc" title="Change Flow Logging Level" >}} for instructions on how to do this.
45+
{{% / alert %}}
46+
4247
## Configure Data Sources
4348

4449
It is necessary to update the Custom Filter inside the dashboards to use the correct data source.
@@ -57,7 +62,7 @@ To do this, follow these steps for all default {{% ctx %}} dashboards imported:
5762
1. Select your [configured Loki data source][] in the *Adhoc Options* > *Data source* drop-down menu, e.g. *grafanacloud-cortex-logs*.
5863
1. Click *Apply*.
5964
1. Click the dashboard name in the breadcrumb at the top left corner of the page to go back to the dashboard.
60-
1. Click the + icon next to the Custom Filter to confirm that a list of available filter options is visible. If Grafana Loki has not received any logs from Promtail there will be no options available for selection.
65+
1. Click the + icon next to the Custom Filter to confirm that a list of available filter options is visible. If Grafana Loki has not received any logs from Grafana Alloy there will be no options available for selection.
6166
1. Repeat steps 2 - 9 for all additional default dashboards.
6267

6368
## Next Steps?
@@ -66,6 +71,6 @@ To do this, follow these steps for all default {{% ctx %}} dashboards imported:
6671

6772
[configured Loki data source]: {{< url path="Cortex.GettingStarted.Cloud.AddObservabilityToInnovation.Grafana.SetupLoki.MainDoc" >}}
6873
[Grafana]: {{< url path="Cortex.GettingStarted.Cloud.AddObservabilityToInnovation.Grafana.SetupGrafana.MainDoc" >}}
69-
[Grafana.Dashboards.zip]: {{< url path="GitHub.Cortex.Observability.3_0_0.GrafanaDashboardsZip" >}}
74+
[Grafana.Dashboards.zip]: {{< url path="GitHub.Cortex.Observability.5_0_0.GrafanaDashboardsZip" >}}
7075
[Loki]: {{< url path="Cortex.GettingStarted.Cloud.AddObservabilityToInnovation.Grafana.SetupLoki.MainDoc" >}}
7176
[Try it Out]: {{< url path="Cortex.GettingStarted.Cloud.AddObservabilityToInnovation.Grafana.TryItOut.MainDoc" >}}

‎content/en/docs/2025.3/getting-started/cloud/add-observability-to-innovation/Grafana/install-alloy/_index.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
---
2+
title: "Install Grafana Alloy"
3+
linkTitle: "Install Grafana Alloy"
4+
description: "Information about installing and configuring Grafana Alloy on the Application Server(s)."
5+
weight: 50
6+
---

‎content/en/docs/2025.3/getting-started/cloud/add-observability-to-innovation/Grafana/install-alloy/configure-alloy.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
title: "Configure Grafana Alloy"
3+
linkTitle: "Configure Grafana Alloy"
4+
description: "Information about configuring Grafana Alloy on the Application Server(s)."
5+
weight: 20
6+
---
7+
8+
# {{% param title %}}
9+
10+
This guide describes how to configure Grafana Alloy on the Application Server(s).
11+
12+
{{% alert type="note" title="Note" %}}These steps must be performed for every Grafana Alloy installation in the cluster.{{% /alert %}}
13+
14+
## Configure Grafana Alloy
15+
16+
1. Open the `config.alloy` configuration file, which is located in the folder alongside the `alloy-installer-windows-amd64.exe` file.
17+
1. Set the `__path__` in the `local.file_match "ApiGateway"` > `path_targets` section to the path of the `Logs` folder for the API Gateway Service, e.g. `"C:/ProgramData/Cortex/API Gateway Service/Logs/**/ServiceFabricHttpEventLog-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]{,_[0-9][0-9][0-9]}.json"`.
18+
1. Set the `__path__` in the `local.file_match "ExecutionService"` > `path_targets` section to the path of the `Logs` folder for the Execution Service, e.g. `"C:/ProgramData/Cortex/Execution Service/Logs/**/*.json"`.
19+
1. Set the Grafana Loki `url` in the `loki.write "default"` > `endpoint` section, which can be found at the end of the file, with the url value noted down during the [Setup Grafana Loki][] steps.
20+
21+
A correct URL should be similar to `https://239948:eyJrIjoiaWVjNzE4MmVjOThkNTgxMMQ5MzIyZjdlMjAyYWY4NWJjO1I1OTc4NSIsIm4iOiJUZXN0S2V5IiwiaWQiOjY4Nzk0MX0=@logs-prod-008.grafana.net/api/prom/push`.
22+
1. Delete the line containing `bearer_token_file` in the `loki.write "default"` > `endpoint` section, which can be found at the end of the file.
23+
1. Save the file.
24+
25+
### Re-Start the Grafana Alloy Service
26+
27+
1. Open `services.msc`.
28+
1. Locate the `Alloy` service.
29+
1. Right click on the service name and select `Restart`. If the service is not already running, select `Start`.
30+
31+
## Next Steps?
32+
33+
1. [Import Dashboards][]
34+
35+
[Import Dashboards]: {{< url path="Cortex.GettingStarted.Cloud.AddObservabilityToInnovation.Grafana.ImportDashboards.MainDoc" >}}
36+
[Setup Grafana Loki]: {{< url path="Cortex.GettingStarted.Cloud.AddObservabilityToInnovation.Grafana.SetupLoki.SetUpGrafanaLoki" >}}

‎content/en/docs/2025.3/getting-started/cloud/add-observability-to-innovation/Grafana/install-alloy/install-alloy.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
title: "Install Grafana Alloy"
3+
linkTitle: "Install Grafana Alloy"
4+
description: "Information about installing Grafana Alloy on the Application Server(s)."
5+
weight: 10
6+
---
7+
8+
# {{% param title %}}
9+
10+
This guide describes how to install Grafana Alloy on the Application Server(s). Please ensure that the [Prerequisites][] have been completed before starting this installation.
11+
12+
## Install Grafana Alloy
13+
14+
1. Download the [Grafana Alloy 1.15.1][] archive.
15+
1. Extract content of the downloaded archive to a suitable location, e.g. `C:\ProgramData\Cortex\Observability\Grafana Alloy`.
16+
1. Download the [Grafana Alloy.zip][] archive and extract its contents alongside the previously extracted Grafana Alloy `alloy-installer-windows-amd64.exe`.
17+
This archive contains the `config.alloy` configuration file and PowerShell scripts to install Grafana Alloy as a Windows service.
18+
1. Run Windows PowerShell as Administrator
19+
1. Change the location to where all the files were extracted to in step 2, e.g. `cd "C:\ProgramData\Cortex\Observability\Grafana Alloy"`.
20+
1. Execute the `.\Install-Alloy.ps1` command to install the downloaded `alloy-installer-windows-amd64.exe` as a service.
21+
1. When prompted, enter the credentials that the Grafana Alloy Service should run as.
22+
1. When prompted to enter the bearer token, leave it blank and press Enter.
23+
24+
## Next Steps?
25+
26+
1. [Configure Grafana Alloy][]
27+
28+
[Configure Grafana Alloy]: {{< url path="Cortex.GettingStarted.Cloud.AddObservabilityToInnovation.Grafana.InstallAlloy.ConfigureAlloy.MainDoc" >}}
29+
[Grafana Alloy 1.15.1]: {{< url path="Grafana.SelfManaged.Downloads.GrafanaAlloy.1_15_1.Windows" >}}
30+
[Grafana Alloy.zip]: {{< url path="GitHub.Cortex.Observability.5_0_0.GrafanaAlloyInstallZip" >}}
31+
[Prerequisites]: {{< url path="Cortex.GettingStarted.OnPremise.AddObservabilityToInnovation.Grafana.Prerequisites" >}}

‎content/en/docs/2025.3/getting-started/cloud/add-observability-to-innovation/Grafana/install-promtail/_index.md‎

Lines changed: 0 additions & 6 deletions
This file was deleted.

0 commit comments

Comments
 (0)