Commit 1e87e94
feat: remove product config & clean up (#790)
* chore(deps): switch stackable-operator to smooth-operator branch
Patches stackable-operator to the operator-rs smooth-operator branch
(provides the v2::config_overrides framework) as the prerequisite for
removing the product-config dependency, mirroring trino-operator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: vendor hadoop-xml and java-properties writers
Copies product-config's writer functions (to_hadoop_xml,
to_java_properties_string) into config::writer, backed directly by the
xml and java-properties crates so output stays byte-identical with the
kuttl ConfigMap snapshots. Removes all product_config::writer usage; the
product-config dependency itself is dropped in a later phase.
Adds unit tests pinning the exact XML/properties on-wire format.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: add controller/build/properties module with ConfigFileName
Introduces the controller/build/properties module structure (mirroring
trino-operator) and a local ConfigFileName enum that replaces the bare
filename string constants as the single source of truth for the
rolegroup ConfigMap keys. Scaffolding for moving per-file rendering out
of the monolithic rolegroup_config_map in the next phase.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: extract hadoop-policy, ssl and security builders
Moves rendering of hadoop-policy.xml, ssl-server.xml, ssl-client.xml and
security.properties out of the monolithic rolegroup_config_map match
into dedicated controller/build/properties modules, each with unit tests
pinning the exact output. Logic is copied verbatim (overrides applied
last), so the rendered ConfigMap is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: extract core-site and hdfs-site builders
Moves the two large inline config renderings out of rolegroup_config_map
into controller/build/properties/{core_site,hdfs_site}, each with unit
tests driven by a shared MINIMAL_HDFS_YAML fixture. Logic is copied
verbatim (overrides applied last), so the rendered ConfigMap is
unchanged. The match arm in rolegroup_config_map is now a thin dispatch.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: inject security.properties DNS cache TTLs in builder
Moves the recommended networkaddress.cache.ttl=30 and
networkaddress.cache.negative.ttl=0 values out of the product-config
properties.yaml and into security_properties::build directly. While
product-config is still active these arrive identically via the override
map applied last, so the rendered file is unchanged; this makes the
builder self-sufficient ahead of removing product-config validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: remove product-config from validation and config building
Replaces the product-config transform/validate steps with native config
merging, mirroring trino-operator:
- validate_cluster no longer takes a ProductConfigManager; it merges the
typed config fragments (HdfsNodeRole::merged_config) and the per-file
configOverrides + envOverrides (role <- role group) into the extended
ValidatedRoleGroupConfig { merged_config, config_overrides, env_overrides }.
- HdfsConfigOverrides now uses typed KeyValueConfigOverrides fields (v2,
deriving Merge) instead of Options; the per-file builders pull their
overrides via resolved_overrides and apply them last.
- Operator-injected values previously provided by properties.yaml /
Configuration impls are now set directly in the builders/validate:
security.properties DNS cache TTLs, dfs.replication, and the rack-aware
net.topology / TOPOLOGY_LABELS (namenode).
- Deletes build_role_properties, the Configuration impls, PropertyNameKind
usage, and the ProductConfigManager plumbing (Ctx, main); the CLI arg is
kept but ignored.
- Filenames now come from the ConfigFileName enum everywhere (discovery,
kerberos, jvm), dropping the *_XML / *_PROPERTIES_FILE constants.
The product-config crate dependency is now unused and removed separately.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: remove the product-config dependency
The product-config crate is no longer used by the operator. Removes it
from the workspace and operator-binary manifests and empties the
config-spec properties.yaml (the schema now lives in the typed CRD and
the per-file builders), mirroring trino-operator. product-config remains
only as a transitive dependency of stackable-operator.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: run linters, regenerate charts, update dependencies
* refactor: extract ConfigMap building into controller/build/config_map
Moves rolegroup_config_map out of hdfs_controller into
controller/build/config_map::build_rolegroup_config_map with its own
Error enum, mirroring trino-operator. The function now takes the
ValidatedCluster and looks up the role group's merged config and
overrides internally instead of receiving them pre-destructured.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: resolve cluster-wide config into ValidatedClusterConfig
Mirrors trino-operator: ValidatedCluster now carries a
ValidatedClusterConfig (name, namespace, dfs_replication, https/kerberos/
authentication/authorization flags, rack awareness, OPA authorization)
resolved once during validation. The standalone hdfs_opa_config field is
folded into it.
The build steps no longer take the raw HdfsCluster:
- hdfs_site/core_site builders and build_rolegroup_config_map consume
&ValidatedClusterConfig / &ValidatedCluster.
- The lower-level HdfsSiteConfigBuilder/CoreSiteConfigBuilder and their
kerberos security_config impls take primitives (a KerberosConfig struct
and bools) to keep the config layer free of controller types.
Behavior-preserving: the flags are computed by the same HdfsCluster
predicates as before, just resolved up-front. The discovery path keeps
using the raw HdfsCluster as it does not go through validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* refactor: add typed ClusterName to ValidatedCluster
Adds a top-level name: ClusterName field to ValidatedCluster (mirroring
trino-operator), constructed and validated in the validate step. The
per-file builders and build_rolegroup_config_map now take the whole
&ValidatedCluster, reading cluster.name and cluster.cluster_config.
product_version is intentionally not added: unlike Trino (integer
versions parsed to u16), HDFS uses semver product versions, so the image
keeps carrying the version string.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: update hashes
* refactor: resolve namenode/journalnode pod refs during validation
Move the pod_refs lookups for namenodes and journalnodes out of
reconcile_hdfs and into validate_cluster, storing the results on
ValidatedCluster. Build steps that already receive the validated
cluster (config_map and hdfs_site) now read the refs from it instead
of taking them as separate parameters.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* style: rustfmt principal_host_part call in kerberos.rs
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: remove left over product config references
* chore: adapted changelog
* refactor: move logging & discovery to build step
* refactor: consume the config-file writer from stackable-operator
Replace the vendored java-properties/Hadoop-XML writer
(rust/operator-binary/src/config/writer.rs) with
stackable_operator::v2::config_file_writer, which hosts this exact file
(moved there verbatim via operator-rs #1217 on the smooth-operator branch;
hdfs's copy was the canonical source). Drop the now-unused java-properties
and xml dependencies.
No behaviour change; rendered output is byte-identical by construction
(same code, new home).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* Apply suggestions from code review
Co-authored-by: Siegfried Weber <mail@siegfriedweber.net>
* fix: simplify iter
* refactor: move rackawareness env to container instead override
* refactor: make ValidatedCluster namespace non optional
* fix: improve ValidatedClusterConfig comment
* fix: move podrefs out of ValidatedCluster
* fix: use Option<T> instead of calculated redundant booleans in ValidatedClusterConfig
* refactor: reorder struct fields in HdfsClusterConfig.
* refactor: consolidate validate_role_group_config
* refactor: use non optional property writers
* fix: improve test formatting
* fix: remove unused pub methods
* fix: remove ownerrefs
* refactor: get infailable ObjectMetaBuilder
* refactor: move ValidatedCluster + siblings to controller/mod.rs
* Use KeyValueConfigOverrides with `BTreeMap<String, String>`
* fix: cleanup errors, constants, duplication
* Regenerate charts
* Use smooth-operator branch for stackable-operator
* Regenerate charts
* refactor: introduce proper framework module split
* fix: restore TOPOLOGY_LABELS envvar behavior
* fix: remove optional EnvVarSet, fix comments
* fix: clippy lint
* refactor: Remove separate merge mechanism
* test(smoke): Add jvmArgumentOverrides
* refactor: Use RoleGroupConfig from stackable-operator
* chore: switch to smooth-operator branch, bump dependencies
* chore: adapt to operator-rs changes (non optional replicas)
* refactor: move k8s resoruces to controller/build/resources, move config, security, operations modules to controller module
* refactor: use v2 types, Port, SecretClassName..
* chore: add uid to ValidatedCluster, add HasName, HasUid, NameIsValidLabelValue
* refactor: use v2 ownerref, service now takes ValdiatedCluster, move helpers from raw cluster to ValidatedCluster
* refactor: make add_pdbs a pure builder and not mutate
* refactor: Introduce ValidatedClusterStatus
* refactor: move computations to build, reduce raw cluster usage
* refactor: extract statefulset from controller
* fix: consolidate method redundant parameters
* fix: reduce RoleGroupRef usages
* feat: use static vector.toml, add test
* fix: use product_logging mod
* chore: use stackable-operator main branch
* chore: use v2 prom scrape label and annotations
* fix: remove raw cluster & last RoleGroupRef
* chore: regenerate
* fix: use ListenerName type
* test: adapt smoke test to logging changes
* refactor: use v2 clusterresources new
* fix: move hardcoded envs to constants, cleanup
* chore: bump stackable-operator to 0.112.0
* chore: use indoc for unit tests
* chore: use more indoc for unit tests
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Andrew Kenworthy <andrew.kenworthy@stackable.tech>
Co-authored-by: Siegfried Weber <mail@siegfriedweber.net>1 parent fd04fee commit 1e87e94
56 files changed
Lines changed: 5052 additions & 6382 deletions
File tree
- deploy
- config-spec
- helm/hdfs-operator/configs
- docs/modules/hdfs/pages/reference
- extra
- rust/operator-binary
- src
- config
- controller
- build
- properties
- product_logging
- resource
- crd
- operations
- security
- tests/templates/kuttl/smoke
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
19 | 19 | | |
20 | 20 | | |
21 | 21 | | |
| 22 | + | |
| 23 | + | |
22 | 24 | | |
23 | 25 | | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
27 | 29 | | |
| 30 | + | |
28 | 31 | | |
29 | 32 | | |
30 | 33 | | |
| |||
0 commit comments