Skip to content

Commit e10a273

Browse files
TechassiTechassi
authored
chore: Update versions for SDP 26.7.0 (#944)
* docs: Update supported versions * test: Update NiFi version in integration tests * test: Remove template gates from upgrade test * test: Adjust flow.py file * test: Add sleep, move assertions This brief sleep ensures that the Pod rotation has started before the assertion (for readyReplicas) is run. Previously, the assertion succeeded before the version upgrade was done and as such, the 07 assertion failed. * test: Add comment in flow.py * docs: Remove NiFi 1 section on Iceberg table page * test: Remove leftover assertions * test: Remove old, unused files * test: Remove old, outdated kubectl cp command
1 parent 79858d5 commit e10a273

57 files changed

Lines changed: 103 additions & 1156 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/modules/nifi/assets/images/put-iceberg-processor.png‎

-85.4 KB
Binary file not shown.

‎docs/modules/nifi/pages/usage_guide/writing-to-iceberg-tables.adoc‎

Lines changed: 0 additions & 88 deletions
Original file line numberDiff line numberDiff line change
@@ -33,91 +33,3 @@ Please read on https://github.com/stackabletech/nifi-iceberg-bundle[its document
3333
You don't need any special configs on the `NiFiCluster` in case you are using S3 and no Kerberos.
3434

3535
HDFS and Kerberos are also supported, please have a look at the https://github.com/stackabletech/nifi-operator/tree/main/tests/templates/kuttl/iceberg[Iceberg integration test] for that.
36-
37-
== NiFi 1
38-
39-
Starting with `1.19.0`, NiFi supports writing to Iceberg tables.
40-
41-
The following example shows an example NiFi setup using the Iceberg integration.
42-
43-
[source,yaml]
44-
----
45-
apiVersion: nifi.stackable.tech/v1alpha1
46-
kind: NifiCluster
47-
metadata:
48-
name: nifi
49-
spec:
50-
clusterConfig:
51-
# ...
52-
extraVolumes:
53-
# Will be mounted at /stackable/userdata/nifi-hive-s3-config/
54-
- name: nifi-hive-s3-config
55-
secret:
56-
secretName: nifi-hive-s3-config
57-
---
58-
apiVersion: v1
59-
kind: Secret
60-
metadata:
61-
name: nifi-hive-s3-config
62-
stringData:
63-
core-site.xml: |
64-
<configuration>
65-
<property>
66-
<name>fs.s3a.aws.credentials.provider</name>
67-
<value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
68-
</property>
69-
70-
<property>
71-
<name>fs.s3a.endpoint</name>
72-
<value>http://minio:9000</value>
73-
</property>
74-
75-
<property>
76-
<name>fs.s3a.access.key</name>
77-
<value>xxx</value>
78-
</property>
79-
80-
<property>
81-
<name>fs.s3a.secret.key</name>
82-
<value>xxx</value>
83-
</property>
84-
85-
<property>
86-
<name>fs.s3a.path.style.access</name>
87-
<value>true</value>
88-
</property>
89-
90-
<property>
91-
<name>fs.s3a.connection.ssl.enabled</name>
92-
<value>false</value>
93-
<description>Enables or disables SSL connections to S3.</description>
94-
</property>
95-
</configuration>
96-
----
97-
98-
Please fill in the correct endpoint, access key and secret key for your S3 store, this is a classic Hadoop config file.
99-
100-
Use e.g. Trino to create a table for Nifi to write into using something like
101-
102-
[source,sql]
103-
----
104-
CREATE SCHEMA IF NOT EXISTS lakehouse.demo WITH (location = 's3a://lakehouse/demo/');
105-
106-
CREATE TABLE IF NOT EXISTS lakehouse.demo.test (
107-
test varchar
108-
);
109-
----
110-
111-
In NiFi you need to create a `HiveCatalogService` first which allows you to access the Hive Metastore storing the Iceberg metadata.
112-
Set `Hive Metastore URI` to something like `thrift://hive-iceberg.default.svc.cluster.local:9083`,
113-
`Default Warehouse Location` to `s3a://lakehouse`
114-
and `Hadoop Configuration Resources` to `/stackable/userdata/nifi-hive-s3-config/core-site.xml`.
115-
116-
Afterwards you can create the `PutIceberg` processor and configure the `HiveCatalogService`.
117-
Also set `Catalog Namespace` to your schema name and the `Table Name`.
118-
119-
For the `File Format` it is recommended to use `PARQUET` or `ORC` rather than `AVRO` for performance reasons, but you can leave it empty or choose your desired format.
120-
121-
You should end up with the following `PutIceberg` processor:
122-
123-
image::put-iceberg-processor.png[]

‎docs/modules/nifi/partials/supported-versions.adoc‎

Lines changed: 1 addition & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -3,9 +3,8 @@
33
// Stackable Platform documentation.
44

55
* 2.9.0
6-
* 2.7.2
6+
* 2.7.2 (Deprecated)
77
** Please note that NiFi starting with 2.7.2 supports Iceberg, but only with S3 and Iceberg REST catalog (no Hive metastore or HDFS support). Please read on xref:nifi:usage_guide/writing-to-iceberg-tables.adoc[] for details.
88
* 2.6.0 (LTS)
9-
* 1.28.1 (Deprecated)
109
1110
For details on how to upgrade your NiFi version, refer to xref:nifi:usage_guide/updating.adoc[].

‎tests/templates/kuttl/cluster-sync-cs-bug/20-install-nifi.yaml.j2‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -166,12 +166,12 @@ metadata:
166166
name: nifi
167167
spec:
168168
image:
169-
{% if test_scenario['values']['nifi-v2'].find(",") > 0 %}
170-
custom: "{{ test_scenario['values']['nifi-v2'].split(',')[1] }}"
171-
productVersion: "{{ test_scenario['values']['nifi-v2'].split(',')[0] }}"
169+
{% if test_scenario['values']['nifi'].find(",") > 0 %}
170+
custom: "{{ test_scenario['values']['nifi'].split(',')[1] }}"
171+
productVersion: "{{ test_scenario['values']['nifi'].split(',')[0] }}"
172172
{% else %}
173173
custom: null
174-
productVersion: "{{ test_scenario['values']['nifi-v2'] }}"
174+
productVersion: "{{ test_scenario['values']['nifi'] }}"
175175
{% endif %}
176176
pullPolicy: IfNotPresent
177177
clusterConfig:

‎tests/templates/kuttl/custom-components-git-sync/30_install-nifi.yaml.j2‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -148,7 +148,7 @@ data:
148148
"bundle": {
149149
"group": "org.apache.nifi",
150150
"artifact": "nifi-prometheus-nar",
151-
"version": "1.28.1"
151+
"version": "2.9.0"
152152
},
153153
"properties": {
154154
"prometheus-reporting-task-metrics-endpoint-port": "8081",
@@ -190,7 +190,7 @@ data:
190190
"bundle": {
191191
"group": "org.apache.nifi",
192192
"artifact": "nifi-standard-nar",
193-
"version": "1.28.1"
193+
"version": "2.9.0"
194194
},
195195
"properties": {
196196
"Regular Expression": "(?s)(^.*$)",
@@ -235,7 +235,7 @@ data:
235235
"bundle": {
236236
"group": "org.apache.nifi",
237237
"artifact": "nifi-standard-nar",
238-
"version": "1.28.1"
238+
"version": "2.9.0"
239239
},
240240
"properties": {
241241
"HTTP Context Map": "fd3b17c4-0196-1000-0000-000076ed786c",
@@ -311,7 +311,7 @@ data:
311311
"bundle": {
312312
"group": "org.apache.nifi",
313313
"artifact": "nifi-standard-nar",
314-
"version": "1.28.1"
314+
"version": "2.9.0"
315315
},
316316
"properties": {
317317
"multipart-request-max-size": "1 MB",
@@ -475,7 +475,7 @@ data:
475475
"bundle": {
476476
"group": "org.apache.nifi",
477477
"artifact": "nifi-http-context-map-nar",
478-
"version": "1.28.1"
478+
"version": "2.9.0"
479479
},
480480
"properties": {
481481
"Request Expiration": "1 min",
@@ -488,7 +488,7 @@ data:
488488
"bundle": {
489489
"group": "org.apache.nifi",
490490
"artifact": "nifi-standard-services-api-nar",
491-
"version": "1.28.1"
491+
"version": "2.9.0"
492492
}
493493
}
494494
],

tests/templates/kuttl/smoke_v1/00-patch-ns.yaml.j2 renamed to tests/templates/kuttl/smoke/00-patch-ns.yaml.j2

File renamed without changes.

tests/templates/kuttl/smoke_v1/00-range-limit.yaml renamed to tests/templates/kuttl/smoke/00-range-limit.yaml

File renamed without changes.

tests/templates/kuttl/smoke_v1/10-assert.yaml.j2 renamed to tests/templates/kuttl/smoke/10-assert.yaml.j2

File renamed without changes.

tests/templates/kuttl/smoke_v1/10-install-vector-aggregator-discovery-configmap.yaml.j2 renamed to tests/templates/kuttl/smoke/10-install-vector-aggregator-discovery-configmap.yaml.j2

File renamed without changes.

tests/templates/kuttl/smoke_v2/20-assert.yaml.j2 renamed to tests/templates/kuttl/smoke/20-assert.yaml.j2

File renamed without changes.

0 commit comments

Comments
 (0)