fix: Tune NiFi startup defaults (election timeout, archive retention) (#936)

* fix: Tune NiFi startup defaults (election timeout, archive retention)

- Remove the operator override of nifi.cluster.flow.election.max.wait.time,
  letting NiFi's upstream default of 5 mins take effect. The previous 1-min
  override was left over from a "for testing" TODO in the operator and may
  have caused flow election to settle on incomplete vote sets in cold-start
  scenarios.

- Set nifi.content.repository.archive.max.retention.period to "3 days".
  Previously empty, which NiFi interprets as Long.MAX_VALUE and disables
  time-based archive purge entirely. Without a time-based ceiling the
  content archive directory can accumulate millions of files, which makes
  the synchronous startup directory scan in FileSystemRepository very slow.
  Users requiring a longer content-replay window can override via
  configOverrides. The provenance audit trail is stored separately and is
  unaffected by this setting.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(smoke_v2): adapt snapshot to default changes

* fix(tests): override nifi.cluster.flow.election.max.wait.time to "10 secs" to avoid test timeouts (5min defaualt).

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: Malte Sander <malte.sander.it@gmail.com>

Author: 3 people

CHANGELOG.md

@@ -17,6 +17,8 @@ All notable changes to this project will be documented in this file.
   Previously, arbitrary keys were silently accepted but ignored ([#921]).
 - Bump `stackable-operator` to 0.110.1 and `kube` to 3.1.0 ([#921]).
 - Internal operator refactoring: introduce dereference() and validate() steps in the reconciler ([#935]).
+- Default `nifi.cluster.flow.election.max.wait.time` to NiFi's upstream value (`5 mins`) instead of the operator's previous `1 mins`. The operator no longer sets this property explicitly; the previous shorter value was left over from a TODO marked as "for testing" and may have caused flow election to settle on incomplete vote sets in cold-start scenarios ([#936]).
+- Set `nifi.content.repository.archive.max.retention.period` to `3 days` (previously empty, which NiFi interprets as `Long.MAX_VALUE` and effectively disables time-based archive purge). Without a time-based ceiling, the content archive can grow to half the content PVC and accumulate millions of files, which makes the synchronous startup directory scan in `FileSystemRepository.initializeRepository` very slow. Users requiring a longer content-replay window can extend via `configOverrides`. The provenance audit trail is independent of this setting and unaffected ([#936]).
 
 ### Fixed
 
@@ -29,6 +31,7 @@ All notable changes to this project will be documented in this file.
 [#924]: https://github.com/stackabletech/nifi-operator/pull/924
 [#928]: https://github.com/stackabletech/nifi-operator/pull/928
 [#935]: https://github.com/stackabletech/nifi-operator/pull/935
+[#936]: https://github.com/stackabletech/nifi-operator/pull/936
 
 ## [26.3.0] - 2026-03-16
 

docs/modules/nifi/pages/usage_guide/operations/graceful-shutdown.adoc

@@ -21,3 +21,7 @@ nifi 2023-11-08 10:52:15,139 INFO [Thread-0] org.apache.nifi.nar.NarAutoLoader N
 nifi 2023-11-08 10:52:15,139 INFO [Thread-0] o.a.n.f.r.CompositeExternalResourceProviderService External Resource Provider Service is stopped
 nifi 2023-11-08 10:52:15,139 INFO [Thread-0] org.apache.nifi.NiFi Application Server shutdown completed
 ----
+
+NOTE: For high-throughput clusters (large numbers of in-flight FlowFiles or high provenance event rates), 5 minutes may not be enough time for NiFi to flush the FlowFile write-ahead log and finalize the provenance index.
+If `SIGKILL` is consistently reached before NiFi can shut down cleanly, repositories are left in a partially-flushed state, which makes the next startup significantly slower (FlowFile WAL recovery and provenance reindex have more work to do).
+Increase `gracefulShutdownTimeout` if you observe `SIGKILL` being issued, or if subsequent restarts take noticeably longer than expected.

docs/modules/nifi/pages/usage_guide/resource-configuration.adoc

@@ -46,3 +46,11 @@ A minimal HA setup consisting of 2 NiFi instances has the following https://kube
 * `18432Mi` persistent storage
 
 You can find the default resource configuration values in the {crd-docs}/nifi.stackable.tech/NifiCluster/v1alpha1/#spec-nodes-config-resources[NifiCluster reference docs {external-link-icon}^].
+
+== Content archive retention
+
+When a FlowFile's content claim is no longer referenced, NiFi moves the file into an `archive/` subdirectory of the content repository rather than deleting it immediately.
+Archived content is what powers *Replay* and *View Content* in the NiFi provenance UI.
+The operator configures both a time-based and a size-based purge threshold; the provenance audit trail itself is stored in a separate repository and is not affected by content archive retention.
+
+If the operator defaults don't fit your workload, for example you need a longer replay window, or you want to disable content archiving entirely, override the relevant `nifi.content.repository.archive.*` properties via xref:usage_guide/overrides.adoc[configOverrides].

rust/operator-binary/src/config/mod.rs

@@ -344,9 +344,19 @@ pub fn build_nifi_properties(
         "nifi.content.repository.directory.default".to_string(),
         NifiRepository::Content.mount_path(),
     );
+    // Cap archived content age so the archive directory stays bounded in
+    // file count. NiFi treats empty as Long.MAX_VALUE, leaving size-based
+    // purge as the only trigger; that lets the archive grow to whatever
+    // half the PVC holds, and the startup directory scan in
+    // FileSystemRepository.initializeRepository scales with file count.
+    // 3 days covers a Friday-incident-investigated-Monday window for
+    // content replay; users with longer requirements can extend via
+    // configOverrides. The percentage-based threshold below acts as a
+    // safety net if write rate outpaces time-based purge.
+    // Also see https://github.com/stackabletech/nifi-operator/issues/354
     properties.insert(
         "nifi.content.repository.archive.max.retention.period".to_string(),
-        "".to_string(),
+        "3 days".to_string(),
     );
     properties.insert(
         "nifi.content.repository.archive.max.usage.percentage".to_string(),
@@ -599,11 +609,6 @@ pub fn build_nifi_properties(
         "nifi.cluster.node.protocol.port".to_string(),
         PROTOCOL_PORT.to_string(),
     );
-    // TODO: set to 1 min for testing (default 5)
-    properties.insert(
-        "nifi.cluster.flow.election.max.wait.time".to_string(),
-        "1 mins".to_string(),
-    );
     properties.insert(
         "nifi.cluster.flow.election.max.candidates".to_string(),
         "".to_string(),

tests/templates/kuttl/cluster-sync-cs-bug/20-install-nifi.yaml.j2

@@ -186,6 +186,8 @@ spec:
       nifi.properties:
         nifi.web.https.sni.required: "false"
         nifi.web.https.sni.host.check: "false"
+        # Quicker startup, and we only have a single node
+        nifi.cluster.flow.election.max.wait.time: "10 secs"
     podOverrides:
       spec:
         initContainers:

tests/templates/kuttl/cluster_operation/20-install-nifi.yaml.j2

@@ -50,6 +50,10 @@ spec:
       gracefulShutdownTimeout: 1m
       logging:
         enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
+    configOverrides:
+      "nifi.properties":
+        # Quicker startup, and we only have a single node
+        "nifi.cluster.flow.election.max.wait.time": "10 secs"
     roleGroups:
       default:
         replicas: 2

tests/templates/kuttl/custom-components-git-sync/30_install-nifi.yaml.j2

@@ -67,6 +67,10 @@ spec:
     config:
       logging:
         enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
+    configOverrides:
+      "nifi.properties":
+        # Quicker startup, and we only have a single node
+        "nifi.cluster.flow.election.max.wait.time": "10 secs"
     roleConfig:
       listenerClass: external-unstable
     podOverrides:

tests/templates/kuttl/external-access/30_nifi.yaml.j2

@@ -53,6 +53,10 @@ spec:
     config:
       logging:
         enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
+    configOverrides:
+      "nifi.properties":
+        # Quicker startup, and we only have a single node
+        "nifi.cluster.flow.election.max.wait.time": "10 secs"
     roleConfig:
       listenerClass: test-external-unstable-$NAMESPACE
     roleGroups:

tests/templates/kuttl/ldap/12-install-nifi.yaml.j2

@@ -54,6 +54,10 @@ spec:
       gracefulShutdownTimeout: 1m
       logging:
         enableVectorAgent: {{ lookup('env', 'VECTOR_AGGREGATOR') | length > 0 }}
+    configOverrides:
+      "nifi.properties":
+        # Quicker startup, and we only have a single node
+        "nifi.cluster.flow.election.max.wait.time": "10 secs"
     roleConfig:
       listenerClass: external-unstable
     roleGroups:

tests/templates/kuttl/logging/04-install-nifi.yaml.j2

@@ -105,6 +105,10 @@ spec:
   nodes:
     config:
       gracefulShutdownTimeout: 1m
+    configOverrides:
+      "nifi.properties":
+        # Quicker startup, and we only have a single node
+        "nifi.cluster.flow.election.max.wait.time": "10 secs"
     roleGroups:
       automatic-log-config:
         replicas: 1