feat: remove product config & clean up (#945)

* chore: point stackable-operator at smooth-operator branch

Add [patch] section to Cargo.toml pointing stackable-operator at the
smooth-operator branch of operator-rs (v0.111.1). No import-path
relocations were needed — all existing paths remain valid on this branch.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: vendor java-properties writer and add ConfigFileName

Vendor the `to_java_properties_string` / `to_hadoop_xml` writers from
`product-config` into `controller::build::properties::writer` so the
operator no longer relies on product-config for config serialization.

Introduce `ConfigFileName` (the enum of NiFi config file names) and
skeleton helpers `defined_entries` / `resolved_overrides` in
`controller::build::properties` — these are dead-code-allowed stubs
that will be wired up in Task 4.

Swap the single existing call site (JVM security properties) to use
the vendored writer; output is byte-identical.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: validate rolegroups via vendored v2 role_utils shim

Introduces a typed, merged-per-rolegroup config path using a vendored
trino-style framework/role_utils.rs shim. Adds NifiRoleGroupConfig type
alias, build_role_group_configs() helper, and role_group_configs field to
ValidatedInputs (strangler pattern — old validated_role_config kept intact).

Also adds Merge impl for NifiConfigOverrides and Ord/Eq/Hash derives to
NifiRole to satisfy generic bounds required by with_validated_config.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: ValidatedInputs -> ValidatedCluster with nested cluster_config

Rename ValidatedInputs to ValidatedCluster and introduce a nested
ValidatedClusterConfig grouping authentication, authorization, and
proxy_hosts. Add a name field (dead_code for now). Add #[derive(Clone)]
to NifiAuthenticationConfig and ResolvedNifiAuthorizationConfig so
reconcile_nifi can clone fields while keeping the validated binding alive
for later tasks.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: per-file build() modules under controller/build/properties

Create one build() module per NiFi config file (bootstrap.conf,
nifi.properties, state-management.xml, security.properties,
login-identity-providers.xml, authorizers.xml) behind the ConfigFileName
enum, sourcing user overrides from the typed NifiRoleGroupConfig instead
of the product-config PropertyNameKind::File maps.

Wire the new builders into build_node_rolegroup_config_map, keeping
validated alive in reconcile_nifi and looking up the typed rg per loop
iteration. Add clustering_backend and sensitive_properties_algorithm to
ValidatedClusterConfig; validate the algorithm at the validate() step.
Move bootstrap.conf tests (now byte-identical) to bootstrap_conf.rs.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: move configmap build into controller/build/config_map.rs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* refactor: source statefulset and git-sync env vars from ValidatedCluster

Replace the product-config validated_role_config map as the env-var source
for the rolegroup loop, git-sync, and statefulset builder with the typed
NifiRoleGroupConfig.env_overrides from ValidatedCluster.role_group_configs.

- Loop now iterates node_role_group_configs (typed BTreeMap) directly
- git-sync gets env vars via env_vars_from_overrides(&rg.env_overrides)
- build_node_rolegroup_statefulset takes &NifiRoleGroupConfig instead of
  &HashMap<PropertyNameKind, BTreeMap<String, String>>
- validated_role_config field is now populated but unread; annotated with
  #[allow(dead_code)] pending removal in Task 7
- Removes: nifi_node_config derivation, MissingRoleGroupConfig error,
  ProductConfigKindNotSpecified error, env_vars_from_rolegroup_config import,
  PropertyNameKind and HashMap imports from controller.rs

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* chore: remove product-config dependency

Drop the `product-config` crate from workspace and operator-binary
Cargo.toml, delete the now-unused `validated_role_config` field and
`validated_product_config()` pipeline, remove ProductConfigManager from
Ctx and main.rs, empty both properties.yaml files to their stub form,
and apply two small nits (use `nifi_role` binding consistently, drop
stale env-var comment).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* test: add unit tests for build modules and validate

Add a test_support helper in controller/build/properties.rs for
constructing a minimal ValidatedCluster without Kubernetes API access
(direct construction chosen over validate::validate() because the latter
requires DereferencedAuthenticationClasses fetched from the API).

Tests added:
- nifi_properties: test_stable_keys_present (HTTPS port, cluster node,
  Kubernetes election impl, sensitive-props algorithm, proxy host wildcard)
- nifi_properties: test_config_override_wins (configOverrides flow-through)
- login_identity_providers: test_build_returns_ok_with_expected_structure
- authorizers: test_build_returns_non_empty_xml_with_authorizers_root

Drive-by rename: BuildProductConfig -> BuildNifiProperties (and its
Snafu selector) in config_map.rs to remove the stale product-config name.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: fix vendored role_utils doc (BTreeMap, nifi)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs: remove product-config from cli and env

* refactor: fix parameters in build configmap

* refactor: move product logging & lint

* chore: regenerate hashes

* chore: adapt changelog

* refactor: consume the config-file writer from stackable-operator

Replace the vendored Java-properties writer
(rust/operator-binary/src/framework/writer.rs) with
stackable_operator::v2::config_file_writer (moved there via operator-rs
#1217 on the smooth-operator branch). NiFi's copy was a java-only subset of
the canonical hdfs writer; the upstream module's additional to_hadoop_xml
simply goes unimported. Drop the now-unused java-properties dependency. The
framework module now only contains role_utils.

The base dependency tag moves from stackable-operator-0.111.0 to 0.111.1,
matching the other operators; cargo only substitutes a [patch] whose package
version matches, and the smooth-operator branch carries 0.111.1.

No behaviour change; rendered .properties output is byte-identical by
construction (same code, new home).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* deps: remove xml-rs dependency

* refactor(1): use NamespaceName and remove optional namespaces

* refactor(2): use uid and metadata.

* refactor: use non optional property writers

* fix: remove error & comment cleanup

* refactor: improve repo handling

* refactor: switch to EnvVarSet

* fix: bump stackable-operator for non optional KeyValueConfigOverrides

* renamed controller, moved validated structs into controller/mod.rs, added validated cluster usage in place of raw cluster in some places

* refactor: use v2 JavaCommonConfig, delete local framework, move jvm.rs to controller/build

* refactor: use v2 owerref, RoleGroupName, labels, ResourceNames

* chore: regenerate

* fix(tests): reduce resources and logging rolegroup length (<16)

* refactor: move k8s resources to build/resource

* fix: cleanup constants

* refactor: move to logging v2, add vector tests

* refactor: move to v2 types, clusterresources, pdbs, container builders

* chore: add prettyassert dependency

* fix: add tests & cleanup

* refactor: remove remaining raw nifi cluster references

* refactor: move gitsync components to ValidatedRoleGroupConfig

* refactor: provide name to ValidatedRoleGroupConfig

* refactor: remove raw nifi cluster from upgrade

* chore: bump dependencies

* refactor: use ConfigMapName, SecretName, ListenerName in CRD

* refactor: use ProductVersion in status

* refactor: use v2 metrics helper

* refactor: use v2 ListenerName

* fix: use STACKABLE_LOG_DIR from framework

* fix: introduce ValidatedRoleConfig

* refactor: remove raw cluster references

* refactor: add objectmeta helper & PLACEHOLDER_LISTENER_ROLE_GROUP

* fix: move configmap to resource folder

* refactor: introduce ValidatedLoggingConfig

* fix: consolidate hard coded constants

* chore: bump dependencies

* fix: adapt to optional replicas

* refactor: use upstream RoleGroupConfig, introduce ValidatedNifiConfig

* refactor: move port constants and use Port type

* refactor: use VolumeName for volumes, tighten constant visibility

* fix: fmt lint

* Revert "fix(tests): reduce resources and logging rolegroup length (<16)"

This reverts commit aa21bd7.

* chore: remove v2/upstream references

* refactor: use v2 types consistently

* refactor: use env_ref and file_ref helpers; consolidate hardcoded constants

* fix: add unit tests

* test: add unit tests

* refactor: consolidate ValidatedReportingTask & ValidatedSensitiveProperties

* refactor: tighten constant visibility

* test: add auth unit fixture

* chore: bump stackable-operator 0.112.0, tokio 1.52, clap 4.6

* fix: rename build sts replicas to effective_replicas (due to upgrade logic) to clarify usage

* fix: undo patching

* linting

* deps: update anyhow for RUSTSEC-2026-0190

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Andrew Kenworthy <andrew.kenworthy@stackable.tech>

Author: 3 people

CHANGELOG.md

@@ -20,6 +20,8 @@ All notable changes to this project will be documented in this file.
 - Default `nifi.cluster.flow.election.max.wait.time` to NiFi's upstream value (`5 mins`) instead of the operator's previous `1 mins`. The operator no longer sets this property explicitly; the previous shorter value was left over from a TODO marked as "for testing" and may have caused flow election to settle on incomplete vote sets in cold-start scenarios ([#936]).
 - Set `nifi.content.repository.archive.max.retention.period` to `3 days` (previously empty, which NiFi interprets as `Long.MAX_VALUE` and effectively disables time-based archive purge). Without a time-based ceiling, the content archive can grow to half the content PVC and accumulate millions of files, which makes the synchronous startup directory scan in `FileSystemRepository.initializeRepository` very slow. Users requiring a longer content-replay window can extend via `configOverrides`. The provenance audit trail is independent of this setting and unaffected ([#936]).
 - test: Bump vector-aggregator to 0.55.0, replace /graphql call with gRPC call ([#940]).
+- BREAKING: Removed product-config machinery. This is a breaking change in terms of configuration.
+  Users relying on the product-config `properties.yaml` file have to set these properties via the CRD ([#945]).
 
 ### Fixed
 
@@ -34,6 +36,7 @@ All notable changes to this project will be documented in this file.
 [#935]: https://github.com/stackabletech/nifi-operator/pull/935
 [#936]: https://github.com/stackabletech/nifi-operator/pull/936
 [#940]: https://github.com/stackabletech/nifi-operator/pull/940
+[#945]: https://github.com/stackabletech/nifi-operator/pull/945
 
 ## [26.3.0] - 2026-03-16