|
| 1 | +# File Log Receiver Configuration |
| 2 | + |
| 3 | +This section describes how to configure file log receiver monitoring using the `environmentWatchConfiguration` JSON configuration file. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## Overview |
| 8 | + |
| 9 | +The file log receiver allows Environment Watch to collect and parse log files from custom sources using the OpenTelemetry Collector. This configuration is intended for Relativity administrators managing Environment Watch on self‑hosted environments who want to extend monitoring to include application‑specific log files (for example, RabbitMQ logs). |
| 10 | + |
| 11 | +By defining log sources in the `openTelemetryOverrides` section of the custom JSON configuration file, administrators can configure log file paths, multiline handling, regex‑based field extraction, and timestamp parsing. |
| 12 | + |
| 13 | +### Assumptions |
| 14 | + |
| 15 | +- The log file is expected to be in **plaintext** format (not JSON), and its path must be accessible from the host where the Environment Watch Windows service is running. |
| 16 | +- The log file must be readable by the Windows account running the Environment Watch Windows service. |
| 17 | +- Each log entry is expected to follow a consistent format that can be matched by the provided regex patterns. |
| 18 | +- The attributes in the `regexPattern` must match the exact field names defined in the source code. For RabbitMQ, the expected field names are: |
| 19 | + |
| 20 | + | Field Name | Log Entry Component | Example Value | |
| 21 | + |--------------------------|----------------------------------------------|----------------------------------------| |
| 22 | + | `rabbitmq_log_date_time` | Timestamp | `2025-10-28 18:20:43.375000-07:00` | |
| 23 | + | `severity` | Log level | `info` | |
| 24 | + | `rabbitmq_pid_node` | First segment of PID `<node.process.serial>` | `0` | |
| 25 | + | `rabbitmq_pid_process` | Second segment of PID | `211` | |
| 26 | + | `rabbitmq_pid_serial` | Third segment of PID | `0` | |
| 27 | + | `message` | Log message body | `ra: starting system coordination` | |
| 28 | + |
| 29 | + **Sample log entry:** |
| 30 | + ```text |
| 31 | + 2025-10-28 18:20:43.375000-07:00 [info] <0.211.0> ra: starting system coordination |
| 32 | + ``` |
| 33 | + |
| 34 | +> [!IMPORTANT] |
| 35 | +> The attributes in the `regexPattern` must exactly match the default field names defined in the source code. Failing to use the correct field names will cause the OpenTelemetry Collector to throw a fatal error and stop working. These field names are also referenced in saved searches and dashboards in Kibana, which may also break if the names do not match. |
| 36 | +
|
| 37 | +--- |
| 38 | + |
| 39 | +## Properties Table |
| 40 | + |
| 41 | +The following table lists the properties used to configure a log source in the custom JSON configuration file. |
| 42 | + |
| 43 | +| Property | Type | Description | |
| 44 | +|--------------------------|---------|------------------------------------------------------------------------------------------------------| |
| 45 | +| `type` | string | Identifies the log source type (e.g., `"rabbitmq"`). | |
| 46 | +| `enabled` | boolean | Enables or disables collection for this log source. | |
| 47 | +| `logFilePath` | string | The full file path to the log file to be monitored (e.g., `"C:\\rabbitmq\\data\\log\\rabbit@localhost.log"`). | |
| 48 | +| `multilineStartPattern` | string | A regex pattern that identifies the start of a new multiline log entry. | |
| 49 | +| `regexPattern` | string | A regex pattern used to extract fields (e.g., timestamp) from each log entry. Supports named capture groups. | |
| 50 | +| `timestampLayout` | string | The timestamp format used to parse the extracted timestamp (e.g., `"%Y-%m-%d %H:%M:%S.%f"`). | |
| 51 | + |
| 52 | +--- |
| 53 | + |
| 54 | +## Configure File Log Receiver |
| 55 | + |
| 56 | +> [!IMPORTANT] |
| 57 | +> The regex patterns shown are examples aligned to known RabbitMQ log formats, and should be validated against the admin’s actual log output before being used in production. Timestamp parsing and field extraction are common failure points, and malformed regex can result in logs being dropped or the collector failing to start. |
| 58 | +
|
| 59 | +The file log receiver is configured in the `openTelemetryOverrides` section of the custom JSON configuration file. The `logSources` array contains one or more log source objects. |
| 60 | +For log sources to monitor, locate `logSources` under the `openTelemetryOverrides` section and update the configuration as shown below. |
| 61 | + |
| 62 | +- `type` : Set to the identifier for the log source (e.g., `"rabbitmq"`). |
| 63 | +- `enabled` : Set to `true` to enable log collection for this source. |
| 64 | +- `logFilePath` : Specify the full path to the log file on the host. |
| 65 | +- `multilineStartPattern` : Provide a regex pattern that matches the beginning of each log entry. |
| 66 | +- `regexPattern` : Provide a regex pattern with named capture groups to extract fields from each log entry. |
| 67 | +- `timestampLayout` : Specify the format string used to parse the extracted timestamp. |
| 68 | + |
| 69 | +**Example** |
| 70 | + |
| 71 | +```json |
| 72 | +{ |
| 73 | + "environmentWatchConfiguration": { |
| 74 | + "monitoring": { |
| 75 | + "instance": { |
| 76 | + "sources": { |
| 77 | + "certificates": {}, |
| 78 | + "windowsServices": {} |
| 79 | + }, |
| 80 | + "otelCollectorYamlFiles": [] |
| 81 | + }, |
| 82 | + "installedProducts": [], |
| 83 | + "hosts": [], |
| 84 | + "openTelemetryOverrides": { |
| 85 | + "logSources": [ |
| 86 | + { |
| 87 | + "type": "rabbitmq", |
| 88 | + "enabled": true, |
| 89 | + "logFilePath": "C:\\rabbitmq\\data\\log\\rabbit@localhost.log", |
| 90 | + "multilineStartPattern": "^\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{6}[+-]\\d{2}:\\d{2}", |
| 91 | + "regexPattern": "^(?P<rabbitmq_log_date_time>\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{6}[+-]\\d{2}:\\d{2}) \\[(?P<severity>[a-z]*)\\] <(?P<rabbitmq_pid_node>\\d+)\\.(?P<rabbitmq_pid_process>\\d+)\\.(?P<rabbitmq_pid_serial>\\d+)>[ \\t]*(?P<message>[\\s\\S]*)$", |
| 92 | + "timestampLayout": "%Y-%m-%d %H:%M:%S.%f%j" |
| 93 | + } |
| 94 | + ] |
| 95 | + } |
| 96 | + }, |
| 97 | + "alertNotificationHandlers": {} |
| 98 | + } |
| 99 | +} |
| 100 | +``` |
| 101 | + |
| 102 | +> [!NOTE] |
| 103 | +> Ensure the `logFilePath` points to a valid log file on the host. If the specified file does not exist, the OpenTelemetry Collector will not be able to collect logs from this source. |
| 104 | +
|
| 105 | +> [!NOTE] |
| 106 | +> The `multilineStartPattern` and `regexPattern` fields use regex syntax. Backslashes must be escaped as `\\` in the JSON configuration file. |
| 107 | +
|
| 108 | +### Default Behavior |
| 109 | + |
| 110 | +Defaults apply at two levels: whether a log source is enabled, and the individual configuration properties used to parse logs. |
| 111 | + |
| 112 | +If a log source is not explicitly configured in the logSources array, it will use its default settings from the source code. For example, if the RabbitMQ log source is not defined in the JSON configuration, it will not be enabled by default since the default enabled value for the RabbitMQ log source is false. To enable it, you must explicitly define it in the logSources array with "enabled": true. |
| 113 | + |
| 114 | +At the property level, if a property is not set or is set to an empty or null value, it falls back to the default value defined in the source code. This applies to the following properties: |
| 115 | + |
| 116 | +| Property | Default Behavior When Empty or Null | |
| 117 | +|--------------------------|--------------------------------------------------------------------------------------------------| |
| 118 | +| `logFilePath` | Falls back to the default log file path defined in the source code for the given log source type. | |
| 119 | +| `multilineStartPattern` | Falls back to the default multiline start pattern defined in the source code. | |
| 120 | +| `regexPattern` | Falls back to the default regex pattern defined in the source code. | |
| 121 | +| `timestampLayout` | Falls back to the default timestamp layout defined in the source code. | |
| 122 | + |
| 123 | +> [!NOTE] |
| 124 | +> Currently, the file log receiver is only supported for RabbitMQ file log receiver. |
| 125 | +
|
| 126 | +> [!NOTE] |
| 127 | +> After updating the custom JSON configuration file, restart the Environment Watch Windows service to apply the changes. |
| 128 | +
|
| 129 | +### Verification in Kibana |
| 130 | + |
| 131 | +- Navigate to **Kibana > Discover**. |
| 132 | +- Select the **logs-\*** data view. |
| 133 | +- Search for `"RabbitMQ configuration found in the shared Json configuration."` to confirm that Environment Watch detected the RabbitMQ file log receiver configuration. |
| 134 | + |
| 135 | + |
| 136 | + |
| 137 | +- If `enabled` is `false`, search for `"No RabbitMQ configuration found in the shared Json config file or rabbitmq was disabled in Json config file. Falling back to default settings."` to confirm that the file log receiver was not enabled or was disabled in the JSON configuration file. |
| 138 | + |
| 139 | + |
| 140 | + |
| 141 | +- Search for `service.name: "server_rabbitmq"` in the **logs-\*** data view. |
| 142 | +- Ensure that log entries from the RabbitMQ log file appear in Kibana. The example below demonstrates how RabbitMQ log entries from the file log receiver are successfully collected and displayed in Kibana. |
| 143 | + |
| 144 | + |
| 145 | + |
| 146 | +> [!NOTE] |
| 147 | +> When `enabled` is `true`, the file log receiver settings are fetched from the custom JSON configuration file. When `enabled` is `false`, the file log receiver falls back to the default settings defined in the source code based on the [Assumptions](#assumptions) made. |
| 148 | +
|
| 149 | +--- |
| 150 | + |
| 151 | +## Troubleshooting |
| 152 | +Refer to the [Troubleshooting Guide](../../../troubleshooting/custom-json-troubleshooting.md) to resolve any custom JSON file log receiver configuration issues. |
0 commit comments