Skip to content

Commit d4a101b

Browse files
labrenbesbernauerNickLarsenNZ
authored
feat: Integrate listener operator (#784)
* add support for listeners * add very basic integration test * add pr number to changelog * add external-access test * chore: Add NiFi 2.4.0 and remove 2.2.0 (#797) add NiFi 2.4.0 and remove 2.2.0 * test: Add test for Apache Iceberg integration (#785) * Clean up smoke test * clean up smoke test part 2 * Add working test :) * Move files * Add and test HDFS functionality * Kerbize HDFS and HMS * Add Kerberos test * Use nightly image * linter * Update Iceberg docs * changelog * Small bumps * Update docs/modules/nifi/pages/usage_guide/writing-to-iceberg-tables.adoc Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com> --------- Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com> * wip: update listener implementation based on implementation for superset operator * use single listener with pvc per rolegroup * update integration tests with crd change * restore iceberg test file * fix listener class in iceberg test * expose https port in headless service * update comment * address feedback from review * remove unused error variants * create headless service name in function * remove unused functions * move listenerClass to roleConfig * use new headless service name in integration tests * move listener constants to listener module * remove duplicate iceberg test * remove hard-coded names * fix integration tests * remove hardcoded role name * improve code quality * set rolegroup label on listener pvcs to none * fix app version label on listener * set rolegroup label on listener pvcs * remove listener class from trino in iceberg test * add note on custom ListenerClasses to docs * remove version argument in reporting task * rename headless service * fix iceberg test * create separate headless services * add missing file * fix iceberg test * use listener scope for tls --------- Co-authored-by: Sebastian Bernauer <sebastian.bernauer@stackable.de> Co-authored-by: Nick <10092581+NickLarsenNZ@users.noreply.github.com>
1 parent 791d835 commit d4a101b

54 files changed

Lines changed: 800 additions & 486 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎CHANGELOG.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ All notable changes to this project will be documented in this file.
77
### Added
88

99
- Add rolling upgrade support for upgrades between NiFi 2 versions ([#771]).
10+
- Added Listener support for NiFi ([#784]).
1011
- Adds new telemetry CLI arguments and environment variables ([#782]).
1112
- Use `--file-log-max-files` (or `FILE_LOG_MAX_FILES`) to limit the number of log files kept.
1213
- Use `--file-log-rotation-period` (or `FILE_LOG_ROTATION_PERIOD`) to configure the frequency of rotation.
@@ -52,6 +53,7 @@ All notable changes to this project will be documented in this file.
5253
[#782]: https://github.com/stackabletech/nifi-operator/pull/782
5354
[#785]: https://github.com/stackabletech/nifi-operator/pull/785
5455
[#787]: https://github.com/stackabletech/nifi-operator/pull/787
56+
[#784]: https://github.com/stackabletech/nifi-operator/pull/784
5557
[#789]: https://github.com/stackabletech/nifi-operator/pull/789
5658
[#793]: https://github.com/stackabletech/nifi-operator/pull/793
5759
[#794]: https://github.com/stackabletech/nifi-operator/pull/794

‎deploy/helm/nifi-operator/crds/crds.yaml‎

Lines changed: 4 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -194,20 +194,6 @@ spec:
194194
description: Allow all proxy hosts by turning off host header validation. See <https://github.com/stackabletech/docker-images/pull/694>
195195
type: boolean
196196
type: object
197-
listenerClass:
198-
default: cluster-internal
199-
description: |-
200-
This field controls which type of Service the Operator creates for this NifiCluster:
201-
202-
* cluster-internal: Use a ClusterIP service
203-
204-
* external-unstable: Use a NodePort service
205-
206-
This is a temporary solution with the goal to keep yaml manifests forward compatible. In the future, this setting will control which [ListenerClass](https://docs.stackable.tech/home/nightly/listener-operator/listenerclass.html) will be used to expose the service, and ListenerClass names will stay the same, allowing for a non-breaking change.
207-
enum:
208-
- cluster-internal
209-
- external-unstable
210-
type: string
211197
sensitiveProperties:
212198
description: These settings configure the encryption of sensitive properties in NiFi processors. NiFi supports encrypting sensitive properties in processors as they are written to disk. You can configure the encryption algorithm and the key to use. You can also let the operator generate an encryption key for you.
213199
properties:
@@ -790,11 +776,15 @@ spec:
790776
x-kubernetes-preserve-unknown-fields: true
791777
roleConfig:
792778
default:
779+
listenerClass: cluster-internal
793780
podDisruptionBudget:
794781
enabled: true
795782
maxUnavailable: null
796783
description: This is a product-agnostic RoleConfig, which is sufficient for most of the products.
797784
properties:
785+
listenerClass:
786+
default: cluster-internal
787+
type: string
798788
podDisruptionBudget:
799789
default:
800790
enabled: true

‎deploy/helm/nifi-operator/templates/roles.yaml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,17 @@ rules:
9090
verbs:
9191
- create
9292
- patch
93+
- apiGroups:
94+
- listeners.stackable.tech
95+
resources:
96+
- listeners
97+
verbs:
98+
- get
99+
- list
100+
- watch
101+
- patch
102+
- create
103+
- delete
93104
- apiGroups:
94105
- {{ include "operator.name" . }}.stackable.tech
95106
resources:

‎docs/modules/nifi/examples/getting_started/getting_started.sh‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -143,15 +143,17 @@ spec:
143143
clusterConfig:
144144
authentication:
145145
- authenticationClass: simple-nifi-users
146-
listenerClass: external-unstable
147146
sensitiveProperties:
148147
keySecret: nifi-sensitive-property-key
149148
autoGenerate: true
150149
zookeeperConfigMapName: simple-nifi-znode
151150
nodes:
151+
roleConfig:
152+
listenerClass: external-unstable
152153
roleGroups:
153154
default:
154155
replicas: 1
156+
155157
EOF
156158
# end::install-nifi[]
157159

‎docs/modules/nifi/examples/getting_started/getting_started.sh.j2‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -143,12 +143,13 @@ spec:
143143
clusterConfig:
144144
authentication:
145145
- authenticationClass: simple-nifi-users
146-
listenerClass: external-unstable
147146
sensitiveProperties:
148147
keySecret: nifi-sensitive-property-key
149148
autoGenerate: true
150149
zookeeperConfigMapName: simple-nifi-znode
151150
nodes:
151+
roleConfig:
152+
listenerClass: external-unstable
152153
roleGroups:
153154
default:
154155
replicas: 1

‎docs/modules/nifi/pages/usage_guide/custom-components.adoc‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -162,12 +162,13 @@ spec:
162162
- name: nifi-processors
163163
configMap:
164164
name: nifi-processors
165-
listenerClass: external-unstable
166165
sensitiveProperties:
167166
keySecret: nifi-sensitive-property-key
168167
autoGenerate: true
169168
zookeeperConfigMapName: simple-nifi-znode
170169
nodes:
170+
roleConfig:
171+
listenerClass: external-unstable
171172
configOverrides:
172173
nifi.properties:
173174
nifi.nar.library.directory.myCustomLibs: /stackable/userdata/nifi-processors/ # <2>
@@ -281,12 +282,13 @@ spec:
281282
- name: nifi-processors
282283
persistentVolumeClaim:
283284
claimName: nifi-processors
284-
listenerClass: external-unstable
285285
sensitiveProperties:
286286
keySecret: nifi-sensitive-property-key
287287
autoGenerate: true
288288
zookeeperConfigMapName: simple-nifi-znode
289289
nodes:
290+
roleConfig:
291+
listenerClass: external-unstable
290292
configOverrides:
291293
nifi.properties:
292294
nifi.nar.library.directory.myCustomLibs: /stackable/userdata/nifi-processors/ # <2>

‎docs/modules/nifi/pages/usage_guide/index.adoc‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,11 +26,12 @@ spec:
2626
- name: nifi-client-certs
2727
secret:
2828
secretName: nifi-client-certs
29-
listenerClass: external-unstable
3029
sensitiveProperties:
3130
keySecret: nifi-sensitive-property-key
3231
autoGenerate: true
3332
nodes:
33+
roleConfig:
34+
listenerClass: external-unstable
3435
roleGroups:
3536
default:
3637
config:

‎docs/modules/nifi/pages/usage_guide/listenerclass.adoc‎

Lines changed: 6 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,19 +1,14 @@
11
= Service exposition with ListenerClasses
22
:description: Configure Apache NiFi service exposure with cluster-internal or external-unstable listener classes.
33

4-
Apache NiFi offers a web UI and an API.
5-
The Operator deploys a service called `<name>` (where `<name>` is the name of the NifiCluster) through which NiFi can be reached.
6-
7-
This service can have either the `cluster-internal` or `external-unstable` type.
8-
`external-stable` is not supported for NiFi at the moment.
9-
Read more about the types in the xref:concepts:service-exposition.adoc[service exposition] documentation at platform level.
10-
11-
This is how the listener class is configured:
4+
The operator deploys a xref:listener-operator:listener.adoc[Listener] for the Node pod.
5+
The listener defaults to only being accessible from within the Kubernetes cluster, but this can be changed by setting `.spec.nodes.roleConfig.listenerClass`:
126

137
[source,yaml]
148
----
159
spec:
16-
clusterConfig:
17-
listenerClass: cluster-internal # <1>
10+
nodes:
11+
roleConfig:
12+
listenerClass: external-unstable # <1>
1813
----
19-
<1> The default `cluster-internal` setting.
14+
<1> Specify one of `external-stable`, `external-unstable`, `cluster-internal` or a custom ListenerClass (the default setting is `cluster-internal`).

‎docs/modules/nifi/pages/usage_guide/monitoring.adoc‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -127,7 +127,7 @@ spec:
127127
- __meta_kubernetes_pod_container_port_number
128128
targetLabel: __address__
129129
replacement: ${1}.${2}.${3}.svc.cluster.local:${4}
130-
regex: (.+);(.+?)(?:-metrics)?;(.+);(.+)
130+
regex: (.+);(.+?)(?:-headless)?;(.+);(.+)
131131
selector:
132132
matchLabels:
133133
prometheus.io/scrape: "true"
@@ -138,4 +138,4 @@ spec:
138138
<1> Authorization via Bearer Token stored in a secret
139139
<2> Relabel \\__address__ to be a FQDN rather then the IP-Address of target pod
140140

141-
NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-metrics` thus we need to regex this suffix.
141+
NOTE: As of xref:listener-operator:listener.adoc[Listener] integration, SDP exposes a Service with `-headless` thus we need to regex this suffix.

‎examples/simple-nifi-cluster.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -51,13 +51,13 @@ spec:
5151
clusterConfig:
5252
authentication:
5353
- authenticationClass: simple-nifi-admin-user
54-
listenerClass: external-unstable
5554
sensitiveProperties:
5655
keySecret: nifi-sensitive-property-key
5756
autoGenerate: true
5857
zookeeperConfigMapName: simple-nifi-znode
5958
nodes:
60-
config:
59+
roleConfig:
60+
listenerClass: external-unstable
6161
roleGroups:
6262
default:
6363
replicas: 1

0 commit comments

Comments
 (0)