Commit abb6225
feat: remove product config (#804)
* feat: add merged_overrides() to AirflowCluster
Computes the merged env and webserver_config.py overrides directly from the
CRD (role <- role-group), replacing the product-config override path.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: remove product-config validation from reconcile
Replaces transform_all_roles_to_config / validate_all_roles_and_groups_config
with direct role iteration plus merged_overrides(). Folds the dereferenced
authentication/authorization objects into ValidatedAirflowCluster so downstream
build steps read them from the validated cluster. The product-config crate is
still used for the Flask config writer (removed in a later step).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: remove product-config CLI parameter
Drops the defunct --product-config section from the commandline reference
and notes the removal of product-config validation in the changelog.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat: remove product-config crate dependency
Vendors the Flask Python-config writer locally (flask_config_writer) and drops
the airflow-operator's direct product-config dependency. The rendered
webserver_config.py is unchanged (the vendored writer is byte-for-byte faithful
to the crate). product-config remains a transitive dependency via
stackable-operator.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: move vendored Flask writer to framework/flask_app_config_writer
Renames the vendored writer back to its upstream name and groups it under a
`framework` module (mirroring the convention in trino-operator), signalling it
as vendored code that is a candidate for a shared crate. The same writer is
still used by superset-operator via the product-config crate. No behaviour
change; the writer body is unchanged from the crate source.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* replace usage of product_config_utils::env_vars_from with inline calls, as done with trino
* changelog update
* refactor: move vendored writer to config/writer, drop framework module
Relocates the vendored Flask config writer from
framework/flask_app_config_writer to config/writer, matching hdfs-operator's
config/writer.rs convention (no operator keeps a vendored writer under
framework/, which trino reserves for v2 upstream mirrors). Converts config.rs
into a config/ module. No behaviour change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: extract webserver_config.py builder into config/webserver_config
Moves the webserver_config.py rendering (defaults + config overrides + the
FILE_HEADER/FILE_FOOTER python blocks + the Flask writer call) out of
build_rolegroup_config_map into a dedicated config::webserver_config::build().
The header/footer key constants and the related error variants move with it.
Drops three debug! traces of intermediate config maps that no longer have a
call site after the extraction; the rendered webserver_config.py is unchanged.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* use constants directly when resolving image
* remove mention of product-config from env-var docs
* refactor: move rolegroup ConfigMap build into controller/build/config_map
Extracts build_rolegroup_config_map out of airflow_controller into a dedicated
controller/build/config_map module with its own error enum, matching the
controller/build/config_map.rs layout in hdfs- and trino-operator. The
controller now wraps it via a single BuildConfigMap error variant; the
ConfigMap-only error variants move into the new module.
No behaviour change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* pass validated cluster instead of individual parameters
* feat: adopt v2 config_overrides; build against the smooth-operator branch
Patches operator-rs to the smooth-operator branch (matching trino- and
hdfs-operator) and actually consumes it: AirflowConfigOverrides.webserver_config_py
now uses stackable_operator::v2::config_overrides::KeyValueConfigOverrides — the
Merge-capable variant trino/hdfs use — instead of the v1 type. Drops the v1
KeyValueOverridesProvider impl and the as_product_config_overrides() call;
merged_overrides reads the override map directly (role <- role-group extend), so
the rendered webserver_config.py is unchanged (39 tests pass).
The CRD gains `nullable: true` on the webserver_config.py override values (v2
allows null to delete a key). Regenerated extra/crds.yaml, Cargo.nix, and
crate-hashes.json.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* docs: make the vendored Flask writer doc operator-agnostic
Neutralises the operator-specific wording in config/writer.rs so the file can be
kept byte-identical between airflow-operator and superset-operator (which vendors
the same writer), making the later move to a shared crate a trivial lift.
Comment-only change.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* refactor: consume the Flask config writer from stackable-operator
Replace the vendored flask writer (rust/operator-binary/src/config/writer.rs)
with stackable_operator::v2::flask_config_writer, which now hosts the same
code (moved there verbatim from this repo via operator-rs #1217, merged into
the smooth-operator branch).
The base dependency tag moves from stackable-operator-0.111.0 to 0.111.1,
matching the other operators. This is required for the [patch] to apply:
cargo only substitutes a patch whose package version matches the dependency,
and the smooth-operator branch carries 0.111.1.
No behaviour change; rendered webserver_config.py output is byte-identical by
construction (same code, new home).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* updated changelog
* remove result as return value as it is now used in calling function
* make webserver_config_py non-optional to bring in line with other operators
* bring addition of log config config maps inline
* regenerate charts
* remove clippy directive and simplify parameters
* add function for multiple callouts
* use emrge instead of extends
* alter structure to attempt to being it mroe in line with e.g. hdfs. Added test for merge/extend difference
* move to v2::role_utils
* add replicas and pod_overrides to ValidatedRoleGroupConfig
* move remaining validated structs to controller
* env_vars_from_overrides to sort deterministically
* regenerate nix
* bump operator-rs to use updated KeyValueConfigOverrides
* regenerate charts
* fix: consolidate ValidatedClusterConfig, use EnvarSet and operator-rs v2 RoleGroupConfig
* refactor: add private metadata to ValidatedCluster to use for ownerreferences
* refactor: move ValidatedCluster structs to controller/mod.rs
* use v2 ownerref, add HasName and HasUid traits N.B. compiled against local/temp op-rs
* refactor: use RoleGroupName, use v2 infallible labels, ResourceNames and reduce usage of RoleGroupRef. N.B. compiled against local op-rs
* bump op-rs and fix ResourceName construction
* regenerate nix
* use metrics_service_name() and v2::product_logging::framework::VectorContainerLogConfig
* refactor: move pdbs to build and return rather than mutate
* refactor: move service to build/resource
* refactor: move listener to build/resource
* refactor: move stateful set and helpers to build/resource
* refactor: remove rolegroupref (1 leftover)
* refactor: switch to v2 logging, add vector test
* refactor: move executor to controller/build
* refactor: remove raw cluster references
* refactor: remove more raw cluster references
* refactor: consolidated graceful shutdown, added db conenection details to ValidatedClusterConfig
* fix: dedup secret name helper
* refactor: switch to v2 metrics name helper
* refactor: add dags_git_sync to ValidatedClusterConfig
* refactor: validate logging upfront
* refactor: make add_config return instead of mutate
* refactor: move graceful shutdown to controller/build
* refactor: merge and move webserver config & envars to controller/build
* refactor: use ConfigMapName, SecretName, ListenerName in CRD
* fix(test): add vector aggregator to opa-interop test
* chore: switch to stackable-operator main branch
* chore: adapt to op-rs changes
* chore: revert rolegroup length changes
* refactor: use VolumeName and Port types
* chore: migrate to new container builder
* chore: migrate to listener volume source builder
* refactor: add ValidatedCluster object meta helper
* fix: add namespace / uid to ValidatedCluster
* fix: introduce ValidatedAirflowConfig, fold logging
* fix: consolidate executor build merging
* fix: consolidate configmap to resource folder, move controller commons to build
* fix: use upstream STACKABLE_LOG_DIR
* fix: introduce ConfigFileName enum
* chore regenerate
* chore: bump stackable-operator 0.111.1 -> 0.112.0, tokio 1.40 -> 1.52, clap 4.5 -> 4.6
* chore: remove test for "missing" fields
* fix: increase scheduler default memory 1GB -> 1.2GB (3.2.2 OOM crashes)
* chore: add scheudler memory increase (1GB -> 1.2GB)
* fix: import "ldap.filter" to fix FAB regression when ldap auth class is used
* fix: remove loop spamming on response status != 200
* fix: bum scheduler memory to 1.5GB to avoid OOM in tests
* fix: split event sources for remote logging 3.2.2 test
* changelog
* clippy warning and comment update
* move db connection resolution out of validate step
* use v2 cluster_resources_new
* git_sync::v1alpha2::GitSyncResources usage inline with nifi
* docs: fix pr tag
* docs: consolidate method comments
* dep: update anyhow for RUSTSEC-2026-0190
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Malte Sander <malte.sander.it@gmail.com>1 parent 0454382 commit abb6225
45 files changed
Lines changed: 4249 additions & 2972 deletions
File tree
- docs/modules/airflow/pages/reference
- extra
- rust/operator-binary
- src
- controller
- build
- properties
- product_logging
- resource
- crd
- operations
- tests/templates/kuttl
- mount-dags-gitsync
- opa-interop
- remote-logging
- smoke
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | 17 | | |
18 | 18 | | |
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
27 | 31 | | |
28 | 32 | | |
29 | 33 | | |
| |||
43 | 47 | | |
44 | 48 | | |
45 | 49 | | |
| 50 | + | |
46 | 51 | | |
47 | 52 | | |
48 | 53 | | |
| |||
0 commit comments