Autogenerate Helm and CRD Vars by jubrad · Pull Request #121 · MaterializeInc/materialize-terraform-self-managed

jubrad · 2025-12-06T04:21:45Z

- Add `materialize_spec_override` to materialize-instance module for arbitrary CRD fields
- Expose `helm_values_override` and `materialize_spec_override` in all cloud examples
- Change default authenticator_kind to Password for all deployments
- Add deepmerge provider for spec merging
- Add Python script to generate Terraform types from upstream CRD/Helm schemas
- Set up pyproject.toml with uv for dependency management
- Add GitHub Action to verify schema sync with upstream Materialize
- Update CONTRIBUTING.md with development setup and type generation docs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Auto generating terraform variables for our CRD and Helm chart is something I've been thinking about for a while and this is a proof of concept for it.

There are many benefits to having an object or set of vars that work with LSP or AI in order to help write code (as opposed to object(any)). And it's also extremely beneficial to be able to set any viable attribute for the helm chart or the materialize CR.

PR has been updated to be a bit more reviewable.
Commit 1 - adds generation script
Commit 2 - adds GH action and docs
Commit 3 - Adds the auto-generated files and adds deprecations
Commit 4 - Adds pytests (yes pytests) for basic validation of our input files to tf_vars
Commit 5 - Adds pytests that use kind and generated tf_vars to deploy a materialize operator via helm and a materialize instance

TODO:

rebase / bring to newest MZ version
Set static object types for kubernetes resources based on type annotations
generate variables automatically in release

jshiwamV · 2025-12-06T13:29:39Z

this looks promising will take a look in detail on monday

alex-hunt-materialize

This is a great idea, but I really don't want us manually parsing and building HCL. Terraform supports json, or we could use a proper HCL parser. I lean toward just converting it all to json, as that opens the door for customers to use their own tools for this too.

I didn't review most of this PR, as it is just a draft. I only did a quick skim of the generator code.

alex-hunt-materialize · 2025-12-08T16:43:10Z


-  # Rollout configuration
-  force_rollout   = var.force_rollout
-  request_rollout = var.request_rollout


We should get consistent about how we handle this stuff. These are removed, while most of the others are marked as deprecated.

yeah, marking this as deprecated is probably for the best.

jubrad · 2025-12-08T16:53:32Z

This is a great idea, but I really don't want us manually parsing and building HCL. Terraform supports json, or we could use a proper HCL parser. I lean toward just converting it all to json, as that opens the door for customers to use their own tools for this too.

Should we use an HCL parser instead?
https://pypi.org/project/python-hcl2/
We could also rewrite the variables file into json, which both python and terraform can parse easily.

Great ideas I'll give both a shot.

jubrad · 2025-12-13T04:29:54Z

@alex-hunt-materialize I went down a rabbit hole with variables in json.tf... the variable type ends up being a string with horrific formatting. We could also use python-hcl2 to output hcl, but it also has shit formatting for complex variables. Hand rolling hcl gen and formatting is potentially the ugliest option to code up, but it generates the cleanest output which I think is worth prioritizing.

alex-hunt-materialize

Overall this looks pretty good. I do worry a bit about how we will test this, as it is a pretty significant change.

alex-hunt-materialize · 2025-12-18T12:42:27Z

+      affinity         = optional(any)
+      defaultResources = optional(any)
+      enabled          = optional(bool)
+      nodeSelector     = optional(any)


Many of these are labeled any. We should at least be able to indicate object(any) for the ones that are objects, even if we can't easily say they are a map of string to string.

I have an update to the auto-generated helm values documentation in MaterializeInc/materialize#34563 which may help with this, if you decide to use the helm-docs generated file.

ok, I think I fixed this now... we'll attempt to use the typenotation k8s/affinity, k8s/tolerations... in the chart parameters to inject some hard coded tfvar types.

alex-hunt-materialize · 2025-12-18T15:09:31Z

+    environmentdExtraEnv = length(var.environmentd_extra_env) > 0 ? [{
+      name = "MZ_SYSTEM_PARAMETER_DEFAULT"
+      value = join(";", [
+        for item in var.environmentd_extra_env :
+        "${item.name}=${item.value}"
+      ])
+    }] : null


Maybe not something to fix in this PR, but this seems like it is wrong and was wrong before this PR. The environmentd_extra_env should not be assumed to be setting a default parameter, but any environment variable.

jubrad · 2026-01-20T05:12:31Z

I'm sure this has been debated and there's good cause for having this in each cloud provider, but naviely it seems like the operator module should just be in the kubernetes module and it makes the tests make more sense.

Introduces generate_terraform_types.py which dynamically generates Terraform variable type definitions from Materialize's upstream schemas: - CRD types from materialize_crd_descriptions.json - Helm parameter types from materialize_operator_chart_parameter.yml The script reads the version from the environmentd_version variable default in the source code and outputs native HCL format (.gen.tf) with proper formatting via terraform fmt. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add check_schema_sync.py to verify generated types match upstream schemas - Add GitHub Actions workflow to run sync check on PRs - Update CONTRIBUTING.md with instructions for regenerating types 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Generated type definitions: - kubernetes/modules/materialize-instance/crd_variables.gen.tf - {aws,azure,gcp}/modules/operator/helm_variables.gen.tf - {aws,azure,gcp}/examples/simple/override_variables.gen.tf Additional changes: - Add helm_values_override and materialize_spec_override variables to operator modules and examples for full customization - Add deprecated force_rollout and request_rollout variables to examples for backwards compatibility, merged into materialize_spec_override - Update materialize-instance module to use the new override variable 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

- Add test-kind.yml workflow to run integration tests on PR - Add kubernetes/modules/operator module with generated helm_values types - Add kind cluster config and backend manifests (postgres, minio) - Add pytest fixtures for cluster lifecycle and terraform deployment - Add integration tests validating operator, instance creation, and pods

jubrad force-pushed the auto-generate-tfvars branch 2 times, most recently from f6c168a to e83a36e Compare December 6, 2025 04:45

alex-hunt-materialize reviewed Dec 8, 2025

View reviewed changes

jubrad force-pushed the auto-generate-tfvars branch 3 times, most recently from 6a3990f to 63cb495 Compare December 13, 2025 04:24

jubrad force-pushed the auto-generate-tfvars branch 3 times, most recently from 83e2a04 to a1e37fb Compare December 13, 2025 05:35

jshiwamV mentioned this pull request Dec 16, 2025

variables.tf file in azure and gcp simple/examples are missing force_rollout and request_rollout #120

Open

alex-hunt-materialize reviewed Dec 18, 2025

View reviewed changes

jubrad commented Jan 20, 2026

View reviewed changes

jubrad force-pushed the auto-generate-tfvars branch 2 times, most recently from 48d3417 to d50dc10 Compare January 20, 2026 19:40

jubrad and others added 5 commits January 30, 2026 16:03

add basic tests for helm value generation

9ada133

jubrad force-pushed the auto-generate-tfvars branch from d50dc10 to 7bc97c1 Compare January 30, 2026 22:04

jubrad changed the title ~~Add Helm and CRD override variables for full customization~~ Autogenerate Helm and CRD Vars Mar 19, 2026

Uh oh!

Conversation

jubrad commented Dec 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jshiwamV commented Dec 6, 2025

Uh oh!

alex-hunt-materialize left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

alex-hunt-materialize Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

jubrad Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

jubrad commented Dec 8, 2025

Uh oh!

jubrad commented Dec 13, 2025

Uh oh!

alex-hunt-materialize left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

alex-hunt-materialize Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

jubrad Jan 30, 2026

Choose a reason for hiding this comment

Uh oh!

alex-hunt-materialize Dec 18, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jubrad Jan 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jubrad commented Dec 6, 2025 •

edited

Loading