Skip to content

Commit b7ca02f

Browse files
siegfriedwebersiegfriedweber
committed
Merge branch 'main' into feat/improve-subject-dn
2 parents 760bf71 + e10a273 commit b7ca02f

61 files changed

Lines changed: 144 additions & 1195 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ All notable changes to this project will be documented in this file.
1919
- Internal operator refactoring: introduce dereference() and validate() steps in the reconciler ([#935]).
2020
- Default `nifi.cluster.flow.election.max.wait.time` to NiFi's upstream value (`5 mins`) instead of the operator's previous `1 mins`. The operator no longer sets this property explicitly; the previous shorter value was left over from a TODO marked as "for testing" and may have caused flow election to settle on incomplete vote sets in cold-start scenarios ([#936]).
2121
- Set `nifi.content.repository.archive.max.retention.period` to `3 days` (previously empty, which NiFi interprets as `Long.MAX_VALUE` and effectively disables time-based archive purge). Without a time-based ceiling, the content archive can grow to half the content PVC and accumulate millions of files, which makes the synchronous startup directory scan in `FileSystemRepository.initializeRepository` very slow. Users requiring a longer content-replay window can extend via `configOverrides`. The provenance audit trail is independent of this setting and unaffected ([#936]).
22+
- test: Bump vector-aggregator to 0.55.0, replace /graphql call with gRPC call ([#940]).
2223

2324
### Fixed
2425

@@ -32,6 +33,7 @@ All notable changes to this project will be documented in this file.
3233
[#928]: https://github.com/stackabletech/nifi-operator/pull/928
3334
[#935]: https://github.com/stackabletech/nifi-operator/pull/935
3435
[#936]: https://github.com/stackabletech/nifi-operator/pull/936
36+
[#940]: https://github.com/stackabletech/nifi-operator/pull/940
3537

3638
## [26.3.0] - 2026-03-16
3739

‎docs/modules/nifi/assets/images/put-iceberg-processor.png‎

-85.4 KB
Binary file not shown.

‎docs/modules/nifi/pages/usage_guide/security.adoc‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,9 @@ spec:
110110

111111
You can follow the xref:tutorials:authentication_with_openldap.adoc[] tutorial to learn how to set up an AuthenticationClass for an LDAP server, as well as consulting the {crd-docs}/authentication.stackable.tech/authenticationclass/v1alpha1/[AuthenticationClass reference {external-link-icon}^].
112112

113+
If your LDAP server uses TLS, the AuthenticationClass also needs to know how to verify the server's certificate.
114+
See xref:concepts:tls-server-verification.adoc[] for how to provide the LDAP server's CA certificate via a SecretClass.
115+
113116
[#authentication-oidc]
114117
=== OIDC
115118

‎docs/modules/nifi/pages/usage_guide/writing-to-iceberg-tables.adoc‎

Lines changed: 0 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -33,91 +33,3 @@ Please read on https://github.com/stackabletech/nifi-iceberg-bundle[its document
3333
You don't need any special configs on the `NiFiCluster` in case you are using S3 and no Kerberos.
3434

3535
HDFS and Kerberos are also supported, please have a look at the https://github.com/stackabletech/nifi-operator/tree/main/tests/templates/kuttl/iceberg[Iceberg integration test] for that.
36-
37-
== NiFi 1
38-
39-
Starting with `1.19.0`, NiFi supports writing to Iceberg tables.
40-
41-
The following example shows an example NiFi setup using the Iceberg integration.
42-
43-
[source,yaml]
44-
----
45-
apiVersion: nifi.stackable.tech/v1alpha1
46-
kind: NifiCluster
47-
metadata:
48-
name: nifi
49-
spec:
50-
clusterConfig:
51-
# ...
52-
extraVolumes:
53-
# Will be mounted at /stackable/userdata/nifi-hive-s3-config/
54-
- name: nifi-hive-s3-config
55-
secret:
56-
secretName: nifi-hive-s3-config
57-
---
58-
apiVersion: v1
59-
kind: Secret
60-
metadata:
61-
name: nifi-hive-s3-config
62-
stringData:
63-
core-site.xml: |
64-
<configuration>
65-
<property>
66-
<name>fs.s3a.aws.credentials.provider</name>
67-
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
68-
</property>
69-
70-
<property>
71-
<name>fs.s3a.endpoint</name>
72-
<value>http://minio:9000</value>
73-
</property>
74-
75-
<property>
76-
<name>fs.s3a.access.key</name>
77-
<value>xxx</value>
78-
</property>
79-
80-
<property>
81-
<name>fs.s3a.secret.key</name>
82-
<value>xxx</value>
83-
</property>
84-
85-
<property>
86-
<name>fs.s3a.path.style.access</name>
87-
<value>true</value>
88-
</property>
89-
90-
<property>
91-
<name>fs.s3a.connection.ssl.enabled</name>
92-
<value>false</value>
93-
<description>Enables or disables SSL connections to S3.</description>
94-
</property>
95-
</configuration>
96-
----
97-
98-
Please fill in the correct endpoint, access key and secret key for your S3 store, this is a classic Hadoop config file.
99-
100-
Use e.g. Trino to create a table for Nifi to write into using something like
101-
102-
[source,sql]
103-
----
104-
CREATE SCHEMA IF NOT EXISTS lakehouse.demo WITH (location = 's3a://lakehouse/demo/');
105-
106-
CREATE TABLE IF NOT EXISTS lakehouse.demo.test (
107-
test varchar
108-
);
109-
----
110-
111-
In NiFi you need to create a `HiveCatalogService` first which allows you to access the Hive Metastore storing the Iceberg metadata.
112-
Set `Hive Metastore URI` to something like `thrift://hive-iceberg.default.svc.cluster.local:9083`,
113-
`Default Warehouse Location` to `s3a://lakehouse`
114-
and `Hadoop Configuration Resources` to `/stackable/userdata/nifi-hive-s3-config/core-site.xml`.
115-
116-
Afterwards you can create the `PutIceberg` processor and configure the `HiveCatalogService`.
117-
Also set `Catalog Namespace` to your schema name and the `Table Name`.
118-
119-
For the `File Format` it is recommended to use `PARQUET` or `ORC` rather than `AVRO` for performance reasons, but you can leave it empty or choose your desired format.
120-
121-
You should end up with the following `PutIceberg` processor:
122-
123-
image::put-iceberg-processor.png[]

‎docs/modules/nifi/partials/supported-versions.adoc‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,8 @@
33
// Stackable Platform documentation.
44

55
* 2.9.0
6-
* 2.7.2
6+
* 2.7.2 (Deprecated)
77
** Please note that NiFi starting with 2.7.2 supports Iceberg, but only with S3 and Iceberg REST catalog (no Hive metastore or HDFS support). Please read on xref:nifi:usage_guide/writing-to-iceberg-tables.adoc[] for details.
88
* 2.6.0 (LTS)
9-
* 1.28.1 (Deprecated)
109
1110
For details on how to upgrade your NiFi version, refer to xref:nifi:usage_guide/updating.adoc[].

‎tests/templates/kuttl/cluster-sync-cs-bug/20-install-nifi.yaml.j2‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -166,12 +166,12 @@ metadata:
166166
name: nifi
167167
spec:
168168
image:
169-
{% if test_scenario['values']['nifi-v2'].find(",") > 0 %}
170-
custom: "{{ test_scenario['values']['nifi-v2'].split(',')[1] }}"
171-
productVersion: "{{ test_scenario['values']['nifi-v2'].split(',')[0] }}"
169+
{% if test_scenario['values']['nifi'].find(",") > 0 %}
170+
custom: "{{ test_scenario['values']['nifi'].split(',')[1] }}"
171+
productVersion: "{{ test_scenario['values']['nifi'].split(',')[0] }}"
172172
{% else %}
173173
custom: null
174-
productVersion: "{{ test_scenario['values']['nifi-v2'] }}"
174+
productVersion: "{{ test_scenario['values']['nifi'] }}"
175175
{% endif %}
176176
pullPolicy: IfNotPresent
177177
clusterConfig:

‎tests/templates/kuttl/custom-components-git-sync/30_install-nifi.yaml.j2‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ data:
148148
"bundle": {
149149
"group": "org.apache.nifi",
150150
"artifact": "nifi-prometheus-nar",
151-
"version": "1.28.1"
151+
"version": "2.9.0"
152152
},
153153
"properties": {
154154
"prometheus-reporting-task-metrics-endpoint-port": "8081",
@@ -190,7 +190,7 @@ data:
190190
"bundle": {
191191
"group": "org.apache.nifi",
192192
"artifact": "nifi-standard-nar",
193-
"version": "1.28.1"
193+
"version": "2.9.0"
194194
},
195195
"properties": {
196196
"Regular Expression": "(?s)(^.*$)",
@@ -235,7 +235,7 @@ data:
235235
"bundle": {
236236
"group": "org.apache.nifi",
237237
"artifact": "nifi-standard-nar",
238-
"version": "1.28.1"
238+
"version": "2.9.0"
239239
},
240240
"properties": {
241241
"HTTP Context Map": "fd3b17c4-0196-1000-0000-000076ed786c",
@@ -311,7 +311,7 @@ data:
311311
"bundle": {
312312
"group": "org.apache.nifi",
313313
"artifact": "nifi-standard-nar",
314-
"version": "1.28.1"
314+
"version": "2.9.0"
315315
},
316316
"properties": {
317317
"multipart-request-max-size": "1 MB",
@@ -475,7 +475,7 @@ data:
475475
"bundle": {
476476
"group": "org.apache.nifi",
477477
"artifact": "nifi-http-context-map-nar",
478-
"version": "1.28.1"
478+
"version": "2.9.0"
479479
},
480480
"properties": {
481481
"Request Expiration": "1 min",
@@ -488,7 +488,7 @@ data:
488488
"bundle": {
489489
"group": "org.apache.nifi",
490490
"artifact": "nifi-standard-services-api-nar",
491-
"version": "1.28.1"
491+
"version": "2.9.0"
492492
}
493493
}
494494
],

‎tests/templates/kuttl/logging/01-install-nifi-vector-aggregator.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ commands:
55
- script: >-
66
helm install nifi-vector-aggregator vector
77
--namespace $NAMESPACE
8-
--version 0.49.0
8+
--version 0.52.0 `# app version 0.55.0`
99
--repo https://helm.vector.dev
1010
--values nifi-vector-aggregator-values.yaml
1111
---

‎tests/templates/kuttl/logging/test_log_aggregation.py‎

Lines changed: 31 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -1,48 +1,44 @@
1-
#!/usr/bin/env python3
2-
import requests
1+
import json
2+
import subprocess
33

44

55
def check_sent_events():
6-
response = requests.post(
7-
'http://nifi-vector-aggregator:8686/graphql',
8-
json={
9-
'query': """
10-
{
11-
transforms(first:100) {
12-
nodes {
13-
componentId
14-
metrics {
15-
sentEventsTotal {
16-
sentEventsTotal
17-
}
18-
}
19-
}
20-
}
21-
}
22-
"""
23-
}
6+
response = subprocess.run(
7+
[
8+
"grpcurl",
9+
"-plaintext",
10+
"-d",
11+
'{"limit": 100}',
12+
"nifi-vector-aggregator:8686",
13+
"vector.observability.v1.ObservabilityService/GetComponents",
14+
],
15+
capture_output=True,
16+
text=True,
17+
check=True, # Raise a CalledProcessError if non-zero return
18+
timeout=20, # seconds
2419
)
20+
result = json.loads(response.stdout)
21+
components = result.get("components", [])
22+
transforms = [
23+
c for c in components if c.get("componentType") == "COMPONENT_TYPE_TRANSFORM"
24+
]
2525

26-
assert response.status_code == 200, \
27-
'Cannot access the API of the vector aggregator.'
26+
assert len(transforms) > 0, "No transform components found"
2827

29-
result = response.json()
30-
31-
transforms = result['data']['transforms']['nodes']
3228
for transform in transforms:
33-
sentEvents = transform['metrics']['sentEventsTotal']
34-
componentId = transform['componentId']
29+
sentEvents = transform["metrics"]["sentEventsTotal"]
30+
componentId = transform["componentId"]
3531

36-
if componentId == 'filteredInvalidEvents':
37-
assert sentEvents is None or \
38-
sentEvents['sentEventsTotal'] == 0, \
39-
'Invalid log events were sent.'
32+
if componentId == "filteredInvalidEvents":
33+
assert sentEvents is None or int(sentEvents) == 0, (
34+
"Invalid log events were sent."
35+
)
4036
else:
41-
assert sentEvents is not None and \
42-
sentEvents['sentEventsTotal'] > 0, \
37+
assert sentEvents is not None and int(sentEvents) > 0, (
4338
f'No events were sent in "{componentId}".'
39+
)
4440

4541

46-
if __name__ == '__main__':
42+
if __name__ == "__main__":
4743
check_sent_events()
48-
print('Test successful!')
44+
print("Test successful!")

tests/templates/kuttl/smoke_v1/00-patch-ns.yaml.j2 renamed to tests/templates/kuttl/smoke/00-patch-ns.yaml.j2

File renamed without changes.

0 commit comments

Comments
 (0)