Skip to content

Commit 2dd1b78

Browse files
philjleedd-agent-integrations-bot[bot]
philjlee
and
dd-agent-integrations-bot[bot]
authored
[AI-6669] NiFi - Add log pipeline and log integration assets (DataDog#23302)
* [AI-6669] Add NiFi log pipeline, assets, manifest, and dashboard Rebased onto master now that the AI-6668 implementation is merged. This PR completes the integration tile by adding assets the implementation PR deliberately left out of scope. Adds: - `manifest.json` with metrics, events, logs, dashboard, saved views - `assets/dashboards/nifi_overview.json` overview dashboard - `assets/dataflows.yaml` declaring nifi-metrics, nifi-events, nifi-logs - `assets/logs/nifi.yaml` pipeline with nested sub-pipelines for app logs and Combined Log Format request logs - `assets/logs/nifi_tests.yaml` with 5 pipeline test cases - `assets/saved_views/` for error logs and HTTP access logs - `images/IMAGES_README.md` media gallery guide - Log config entries in `spec.yaml` and `conf.yaml.example` for all four NiFi log files (app, user, bootstrap, request) - README log collection section with multiline stack trace guidance Drops the earlier `service_checks.json` and its manifest reference — the check only emits the `nifi.can_connect` gauge, not a service check, so the README already documents connectivity there. * Update dependency resolution * Address review feedback on NiFi dashboard and saved view - Fix GC widget query to use `sum:...as_rate()` for correct rate semantics on the monotonic_count metric (was `avg:` which misrepresents the rate). - Expand the overview dashboard to cover cluster health, bytes queued, processor status breakdown, and top connections by backpressure, so the dashboard reflects the full scope the check advertises. - Fix the error-logs saved view query: include `warning` status so 4xx HTTP request logs (tagged `status:warning` by the pipeline) surface too. * Fix CI validation failures for NiFi log pipeline - Add missing standard attribute facets (URL host/port/scheme, useragent browser/device/os families) and fix sort order - Update request log test expectations to match actual pipeline user-agent parser output - Add changelog entry for PR DataDog#23302 * Remove dashboard, manifest, and images from repo These assets belong in Developer Platform, not integrations-core. --------- Co-authored-by: dd-agent-integrations-bot[bot] <dd-agent-integrations-bot[bot]@users.noreply.github.com>
1 parent 7ddb917 commit 2dd1b78

9 files changed

Lines changed: 446 additions & 0 deletions

File tree

‎nifi/README.md‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,56 @@ No additional installation is needed on your server.
3131
3232
3. [Restart the Agent][5].
3333
34+
#### Log collection
35+
36+
1. Collecting logs is disabled by default in the Datadog Agent. Enable it in your `datadog.yaml` file:
37+
38+
```yaml
39+
logs_enabled: true
40+
```
41+
42+
2. Uncomment and edit the logs configuration block in your `nifi.d/conf.yaml` file. NiFi produces several log files; configure the ones relevant to your environment:
43+
44+
```yaml
45+
logs:
46+
- type: file
47+
path: /opt/nifi/logs/nifi-app.log
48+
source: nifi
49+
service: nifi
50+
- type: file
51+
path: /opt/nifi/logs/nifi-user.log
52+
source: nifi
53+
service: nifi
54+
- type: file
55+
path: /opt/nifi/logs/nifi-bootstrap.log
56+
source: nifi
57+
service: nifi
58+
- type: file
59+
path: /opt/nifi/logs/nifi-request.log
60+
source: nifi
61+
service: nifi
62+
tags:
63+
- "log_type:request"
64+
```
65+
66+
The `log_type:request` tag on the request log entry routes HTTP access logs through a dedicated parsing pipeline that extracts standard HTTP attributes (method, status code, URL path, client IP).
67+
68+
NiFi is a Java application that produces multiline stack traces. To aggregate them into single log events, add a `log_processing_rules` entry to the application log:
69+
70+
```yaml
71+
logs:
72+
- type: file
73+
path: /opt/nifi/logs/nifi-app.log
74+
source: nifi
75+
service: nifi
76+
log_processing_rules:
77+
- type: multi_line
78+
name: java_stack_trace
79+
pattern: \d{4}-\d{2}-\d{2}
80+
```
81+
82+
3. [Restart the Agent][5].
83+
3484
### Validation
3585

3686
[Run the Agent's status subcommand][6] and look for `nifi` under the Checks section.

‎nifi/assets/configuration/spec.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,3 +87,17 @@ files:
8787
path: /opt/nifi/logs/nifi-app.log
8888
source: nifi
8989
service: nifi
90+
- type: file
91+
path: /opt/nifi/logs/nifi-user.log
92+
source: nifi
93+
service: nifi
94+
- type: file
95+
path: /opt/nifi/logs/nifi-bootstrap.log
96+
source: nifi
97+
service: nifi
98+
- type: file
99+
path: /opt/nifi/logs/nifi-request.log
100+
source: nifi
101+
service: nifi
102+
tags:
103+
- "log_type:request"

‎nifi/assets/dataflows.yaml‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
provides:
2+
- id: nifi-metrics
3+
always_on: true
4+
data_type: metrics
5+
direction: inbound
6+
- id: nifi-events
7+
always_on: true
8+
data_type: events
9+
direction: inbound
10+
- id: nifi-logs
11+
always_on: false
12+
data_type: logs
13+
direction: inbound

‎nifi/assets/logs/nifi.yaml‎

Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
id: nifi
2+
metric_id: nifi
3+
backend_only: false
4+
facets:
5+
- groups:
6+
- Web Access
7+
name: Method
8+
path: http.method
9+
source: log
10+
- groups:
11+
- Web Access
12+
name: Status Code
13+
path: http.status_code
14+
source: log
15+
- groups:
16+
- Web Access
17+
name: URL Host
18+
path: http.url_details.host
19+
source: log
20+
- groups:
21+
- Web Access
22+
name: URL Path
23+
path: http.url_details.path
24+
source: log
25+
- groups:
26+
- Web Access
27+
name: URL Port
28+
path: http.url_details.port
29+
source: log
30+
- groups:
31+
- Web Access
32+
name: URL scheme
33+
path: http.url_details.scheme
34+
source: log
35+
- groups:
36+
- Web Access
37+
name: Browser
38+
path: http.useragent_details.browser.family
39+
source: log
40+
- groups:
41+
- Web Access
42+
name: Device
43+
path: http.useragent_details.device.family
44+
source: log
45+
- groups:
46+
- Web Access
47+
name: OS
48+
path: http.useragent_details.os.family
49+
source: log
50+
- groups:
51+
- Source Code
52+
name: Logger Name
53+
path: logger.name
54+
source: log
55+
- groups:
56+
- Source Code
57+
name: Thread Name
58+
path: logger.thread_name
59+
source: log
60+
- groups:
61+
- Web Access
62+
name: Client IP
63+
path: network.client.ip
64+
source: log
65+
pipeline:
66+
type: pipeline
67+
name: NiFi
68+
enabled: true
69+
filter:
70+
query: source:nifi
71+
processors:
72+
- type: pipeline
73+
name: NiFi application logs
74+
enabled: true
75+
filter:
76+
query: -log_type:request
77+
processors:
78+
- type: grok-parser
79+
name: Parsing NiFi application logs
80+
enabled: true
81+
source: message
82+
samples:
83+
- "2026-03-20 18:08:33,065 INFO [main] o.a.nifi.NiFi NiFi PID is 1"
84+
- "2026-03-20 18:08:33,065 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.PutFile PutFile[id=abc] Failed to write"
85+
- "2026-03-20 18:08:34,100 WARN [Flow Service Tasks Thread-1] o.a.n.c.s.impl.WriteAheadFlowFileRepository Encountered non-critical error"
86+
grok:
87+
supportRules: |
88+
_date %{date("yyyy-MM-dd HH:mm:ss,SSS"):timestamp}
89+
_level %{word:level}
90+
_thread %{regex("[^]]*"):logger.thread_name}
91+
_logger %{notSpace:logger.name}
92+
matchRules: |
93+
nifi_app %{_date} %{_level} \[%{_thread}\] %{_logger} %{data:message}
94+
- type: date-remapper
95+
name: Define `timestamp` as the official date of the log
96+
enabled: true
97+
sources:
98+
- timestamp
99+
- type: status-remapper
100+
name: Define `level` as the official status of the log
101+
enabled: true
102+
sources:
103+
- level
104+
- type: message-remapper
105+
name: Define `message` as the official message of the log
106+
enabled: true
107+
sources:
108+
- message
109+
- type: pipeline
110+
name: NiFi HTTP request logs
111+
enabled: true
112+
filter:
113+
query: log_type:request
114+
processors:
115+
- type: grok-parser
116+
name: Parsing NiFi HTTP request logs
117+
enabled: true
118+
source: message
119+
samples:
120+
- '192.168.1.100 - admin [20/Mar/2026:18:08:33 +0000] "GET /nifi-api/flow/status HTTP/1.1" 200 1234 "-" "python-requests/2.31.0"'
121+
- '10.0.0.50 - - [20/Mar/2026:18:10:45 +0000] "POST /nifi-api/process-groups/root/process-groups HTTP/1.1" 404 892 "https://localhost:8443/nifi/" "Mozilla/5.0"'
122+
grok:
123+
supportRules: |
124+
_auth %{notSpace:http.auth:nullIf("-")}
125+
_bytes_written %{integer:network.bytes_written}
126+
_client_ip %{ipOrHost:network.client.ip}
127+
_version HTTP\/%{regex("\\d+\\.\\d+"):http.version}
128+
_url %{notSpace:http.url}
129+
_ident %{notSpace:http.ident:nullIf("-")}
130+
_user_agent %{regex("[^\\\"]*"):http.useragent}
131+
_referer %{notSpace:http.referer}
132+
_status_code %{integer:http.status_code}
133+
_method %{word:http.method}
134+
_date_access %{date("dd/MMM/yyyy:HH:mm:ss Z"):date_access}
135+
matchRules: |
136+
access.common %{_client_ip} %{_ident} %{_auth} \[%{_date_access}\] "(?>%{_method} |)%{_url}(?> %{_version}|)" %{_status_code} (?>%{_bytes_written}|-)
137+
138+
access.combined %{access.common} "%{_referer}" "%{_user_agent}"
139+
- type: url-parser
140+
name: Extract URL details
141+
enabled: true
142+
sources:
143+
- http.url
144+
target: http.url_details
145+
- type: user-agent-parser
146+
name: Extract user agent details
147+
enabled: true
148+
sources:
149+
- http.useragent
150+
target: http.useragent_details
151+
encoded: false
152+
- type: date-remapper
153+
name: Define `date_access` as the official date of the log
154+
enabled: true
155+
sources:
156+
- date_access
157+
- type: category-processor
158+
name: Categorize status code
159+
enabled: true
160+
categories:
161+
- filter:
162+
query: "@http.status_code:[200 TO 299]"
163+
name: OK
164+
- filter:
165+
query: "@http.status_code:[300 TO 399]"
166+
name: notice
167+
- filter:
168+
query: "@http.status_code:[400 TO 499]"
169+
name: warning
170+
- filter:
171+
query: "@http.status_code:[500 TO 599]"
172+
name: error
173+
target: http.status_category
174+
- type: status-remapper
175+
name: Define `http.status_category` as the official status of the log
176+
enabled: true
177+
sources:
178+
- http.status_category

‎nifi/assets/logs/nifi_tests.yaml‎

Lines changed: 117 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,117 @@
1+
id: "nifi"
2+
tests:
3+
-
4+
sample: "2026-03-20 18:08:33,065 INFO [main] o.a.nifi.NiFi NiFi PID is 1"
5+
result:
6+
custom:
7+
level: "INFO"
8+
logger:
9+
name: "o.a.nifi.NiFi"
10+
thread_name: "main"
11+
timestamp: 1774030113065
12+
message: "NiFi PID is 1"
13+
status: "info"
14+
tags:
15+
- "source:LOGS_SOURCE"
16+
timestamp: 1774030113065
17+
-
18+
sample: "2026-03-20 18:08:33,065 ERROR [Timer-Driven Process Thread-1] o.a.n.p.standard.PutFile PutFile[id=abc] Failed to write"
19+
result:
20+
custom:
21+
level: "ERROR"
22+
logger:
23+
name: "o.a.n.p.standard.PutFile"
24+
thread_name: "Timer-Driven Process Thread-1"
25+
timestamp: 1774030113065
26+
message: "PutFile[id=abc] Failed to write"
27+
status: "error"
28+
tags:
29+
- "source:LOGS_SOURCE"
30+
timestamp: 1774030113065
31+
-
32+
sample: "2026-03-20 18:08:34,100 WARN [Flow Service Tasks Thread-1] o.a.n.c.s.impl.WriteAheadFlowFileRepository Encountered non-critical error"
33+
result:
34+
custom:
35+
level: "WARN"
36+
logger:
37+
name: "o.a.n.c.s.impl.WriteAheadFlowFileRepository"
38+
thread_name: "Flow Service Tasks Thread-1"
39+
timestamp: 1774030114100
40+
message: "Encountered non-critical error"
41+
status: "warn"
42+
tags:
43+
- "source:LOGS_SOURCE"
44+
timestamp: 1774030114100
45+
-
46+
sample: '192.168.1.100 - admin [20/Mar/2026:18:08:33 +0000] "GET /nifi-api/flow/status HTTP/1.1" 200 1234 "-" "python-requests/2.31.0"'
47+
tags:
48+
- "log_type:request"
49+
result:
50+
custom:
51+
date_access: 1774030113000
52+
http:
53+
auth: "admin"
54+
method: "GET"
55+
referer: "-"
56+
status_category: "OK"
57+
status_code: 200
58+
url: "/nifi-api/flow/status"
59+
url_details:
60+
path: "/nifi-api/flow/status"
61+
useragent: "python-requests/2.31.0"
62+
useragent_details:
63+
browser:
64+
family: "Python Requests"
65+
major: "2"
66+
minor: "31"
67+
device:
68+
category: "Other"
69+
family: "Other"
70+
os:
71+
family: "Other"
72+
version: "1.1"
73+
network:
74+
bytes_written: 1234
75+
client:
76+
ip: "192.168.1.100"
77+
message: '192.168.1.100 - admin [20/Mar/2026:18:08:33 +0000] "GET /nifi-api/flow/status HTTP/1.1" 200 1234 "-" "python-requests/2.31.0"'
78+
status: "ok"
79+
tags:
80+
- "source:LOGS_SOURCE"
81+
- "log_type:request"
82+
timestamp: 1774030113000
83+
-
84+
sample: '10.0.0.50 - - [20/Mar/2026:18:10:45 +0000] "POST /nifi-api/process-groups/root/process-groups HTTP/1.1" 404 892 "https://localhost:8443/nifi/" "Mozilla/5.0"'
85+
tags:
86+
- "log_type:request"
87+
result:
88+
custom:
89+
date_access: 1774030245000
90+
http:
91+
method: "POST"
92+
referer: "https://localhost:8443/nifi/"
93+
status_category: "warning"
94+
status_code: 404
95+
url: "/nifi-api/process-groups/root/process-groups"
96+
url_details:
97+
path: "/nifi-api/process-groups/root/process-groups"
98+
useragent: "Mozilla/5.0"
99+
useragent_details:
100+
browser:
101+
family: "Other"
102+
device:
103+
category: "Other"
104+
family: "Other"
105+
os:
106+
family: "Other"
107+
version: "1.1"
108+
network:
109+
bytes_written: 892
110+
client:
111+
ip: "10.0.0.50"
112+
message: '10.0.0.50 - - [20/Mar/2026:18:10:45 +0000] "POST /nifi-api/process-groups/root/process-groups HTTP/1.1" 404 892 "https://localhost:8443/nifi/" "Mozilla/5.0"'
113+
status: "warn"
114+
tags:
115+
- "source:LOGS_SOURCE"
116+
- "log_type:request"
117+
timestamp: 1774030245000

0 commit comments

Comments
 (0)