Skip to content

Commit e83a36e

Browse files
jubradclaude
jubrad
and
claude
committed
Add schema sync check script
Move sync checking logic from workflow into a reusable Python script that verifies generated Terraform types match upstream schemas. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
1 parent b7d877a commit e83a36e

12 files changed

Lines changed: 368 additions & 823 deletions

File tree

‎.github/workflows/check-schema-sync.yml‎

Lines changed: 2 additions & 86 deletions
Original file line numberDiff line numberDiff line change
@@ -4,9 +4,6 @@ on:
44
merge_group:
55
pull_request:
66
branches: [main]
7-
schedule:
8-
# Run weekly to detect upstream schema changes
9-
- cron: "0 0 * * 0"
107
workflow_dispatch:
118

129
jobs:
@@ -28,86 +25,5 @@ jobs:
2825
- name: Install dependencies
2926
run: uv sync
3027

31-
- name: Extract default Materialize version
32-
id: get-version
33-
run: |
34-
# Extract version from the materialize-instance module (marked with # META: mz version)
35-
VERSION=$(grep -E "default.*=.*\"v[0-9]+\.[0-9]+\.[0-9]+\".*# META: mz version" \
36-
kubernetes/modules/materialize-instance/variables.tf | \
37-
grep -oE "v[0-9]+\.[0-9]+\.[0-9]+" | head -1)
38-
echo "version=$VERSION" >> $GITHUB_OUTPUT
39-
echo "Detected Materialize version: $VERSION"
40-
41-
- name: Validate CRD schema is accessible
42-
run: |
43-
VERSION="${{ steps.get-version.outputs.version }}"
44-
CRD_URL="https://raw.githubusercontent.com/MaterializeInc/materialize/${VERSION}/doc/user/data/self_managed/materialize_crd_descriptions.json"
45-
echo "Checking CRD schema at: $CRD_URL"
46-
47-
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$CRD_URL")
48-
if [ "$HTTP_STATUS" != "200" ]; then
49-
echo "::error::CRD schema not found at $CRD_URL (HTTP $HTTP_STATUS)"
50-
echo "This may indicate the Materialize version $VERSION does not exist or the schema path has changed."
51-
exit 1
52-
fi
53-
echo "CRD schema accessible"
54-
55-
- name: Validate Helm values schema is accessible
56-
run: |
57-
VERSION="${{ steps.get-version.outputs.version }}"
58-
HELM_URL="https://raw.githubusercontent.com/MaterializeInc/materialize/${VERSION}/misc/helm-charts/operator/values.yaml"
59-
echo "Checking Helm values at: $HELM_URL"
60-
61-
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$HELM_URL")
62-
if [ "$HTTP_STATUS" != "200" ]; then
63-
echo "::error::Helm values not found at $HELM_URL (HTTP $HTTP_STATUS)"
64-
echo "This may indicate the Materialize version $VERSION does not exist or the values path has changed."
65-
exit 1
66-
fi
67-
echo "Helm values schema accessible"
68-
69-
- name: Run type generation script (dry run)
70-
run: |
71-
VERSION="${{ steps.get-version.outputs.version }}"
72-
echo "Running type generation script for version $VERSION..."
73-
uv run python scripts/generate_terraform_types.py --version "$VERSION" --output simplified > /dev/null
74-
echo "Type generation script completed successfully"
75-
76-
- name: Check for new CRD fields
77-
run: |
78-
VERSION="${{ steps.get-version.outputs.version }}"
79-
CRD_URL="https://raw.githubusercontent.com/MaterializeInc/materialize/${VERSION}/doc/user/data/self_managed/materialize_crd_descriptions.json"
80-
81-
# Fetch current CRD fields
82-
curl -s "$CRD_URL" | python3 -c "
83-
import json
84-
import sys
85-
86-
data = json.load(sys.stdin)
87-
88-
# Find MaterializeSpec
89-
for item in data:
90-
if item[0] == 'MaterializeSpec':
91-
fields = item[1]
92-
print('MaterializeSpec fields:')
93-
for field in fields:
94-
deprecated = ' (DEPRECATED)' if field.get('deprecated', False) else ''
95-
required = ' (required)' if field.get('required', False) else ''
96-
print(f\" - {field['name']}: {field['type']}{required}{deprecated}\")
97-
break
98-
"
99-
100-
- name: Summary
101-
run: |
102-
VERSION="${{ steps.get-version.outputs.version }}"
103-
echo "## Schema Sync Check Summary" >> $GITHUB_STEP_SUMMARY
104-
echo "" >> $GITHUB_STEP_SUMMARY
105-
echo "- **Materialize Version:** $VERSION" >> $GITHUB_STEP_SUMMARY
106-
echo "- **CRD Schema:** Accessible" >> $GITHUB_STEP_SUMMARY
107-
echo "- **Helm Values:** Accessible" >> $GITHUB_STEP_SUMMARY
108-
echo "- **Type Generation:** Successful" >> $GITHUB_STEP_SUMMARY
109-
echo "" >> $GITHUB_STEP_SUMMARY
110-
echo "To regenerate types after a version bump, run:" >> $GITHUB_STEP_SUMMARY
111-
echo "\`\`\`bash" >> $GITHUB_STEP_SUMMARY
112-
echo "uv run python scripts/generate_terraform_types.py --version $VERSION" >> $GITHUB_STEP_SUMMARY
113-
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
28+
- name: Check schema sync
29+
run: uv run python scripts/check_schema_sync.py --show-fields

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -29,6 +29,9 @@ tests/.terraform.lock.hcl
2929
# macOS
3030
.DS_Store
3131

32+
# Python
33+
__pycache__/
34+
3235
# IDE
3336
.idea/
3437
.vscode/

‎CONTRIBUTING.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,22 @@ uv run python scripts/generate_terraform_types.py --output full
6565

6666
The script fetches the CRD field descriptions and Helm values from the Materialize repository and generates corresponding Terraform variable type definitions.
6767

68+
### Checking Schema Sync
69+
70+
To verify that the generated Terraform types are in sync with upstream Materialize schemas:
71+
72+
```bash
73+
uv run python scripts/check_schema_sync.py
74+
75+
# Show CRD fields for debugging
76+
uv run python scripts/check_schema_sync.py --show-fields
77+
78+
# Check against a specific version
79+
uv run python scripts/check_schema_sync.py --version v26.1.0
80+
```
81+
82+
This check runs automatically in CI via the `check-schema-sync` workflow.
83+
6884
## Development Process
6985

7086
1. Clone the repository

‎aws/examples/simple/main.tf‎

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -330,10 +330,6 @@ module "materialize_instance" {
330330
metadata_backend_url = local.metadata_backend_url
331331
persist_backend_url = local.persist_backend_url
332332

333-
# Rollout configuration
334-
force_rollout = var.force_rollout
335-
request_rollout = var.request_rollout
336-
337333
# Password authentication for the Materialize instance
338334
external_login_password_mz_system = random_password.external_login_password_mz_system.result
339335
authenticator_kind = "Password"

‎aws/examples/simple/override_variables.tf‎

Lines changed: 9 additions & 119 deletions
Original file line numberDiff line numberDiff line change
@@ -33,125 +33,15 @@ variable "helm_values_override" {
3333
}))
3434
clusters = optional(object({
3535
swap_enabled = optional(bool)
36-
sizes = optional(object({
37-
mz_probe = optional(object({
38-
workers = optional(number)
39-
scale = optional(number)
40-
cpu_exclusive = optional(bool)
41-
cpu_limit = optional(number)
42-
credits_per_hour = optional(string)
43-
disk_limit = optional(string)
44-
memory_limit = optional(string)
45-
}))
46-
"25cc" = optional(object({
47-
workers = optional(number)
48-
scale = optional(number)
49-
cpu_exclusive = optional(bool)
50-
cpu_limit = optional(number)
51-
credits_per_hour = optional(string)
52-
disk_limit = optional(string)
53-
memory_limit = optional(string)
54-
}))
55-
"50cc" = optional(object({
56-
workers = optional(number)
57-
scale = optional(number)
58-
cpu_exclusive = optional(bool)
59-
cpu_limit = optional(number)
60-
credits_per_hour = optional(string)
61-
disk_limit = optional(string)
62-
memory_limit = optional(string)
63-
}))
64-
"100cc" = optional(object({
65-
workers = optional(number)
66-
scale = optional(number)
67-
cpu_exclusive = optional(bool)
68-
cpu_limit = optional(number)
69-
credits_per_hour = optional(string)
70-
disk_limit = optional(string)
71-
memory_limit = optional(string)
72-
}))
73-
"200cc" = optional(object({
74-
workers = optional(number)
75-
scale = optional(number)
76-
cpu_exclusive = optional(bool)
77-
cpu_limit = optional(number)
78-
credits_per_hour = optional(string)
79-
disk_limit = optional(string)
80-
memory_limit = optional(string)
81-
}))
82-
"300cc" = optional(object({
83-
workers = optional(number)
84-
scale = optional(number)
85-
cpu_exclusive = optional(bool)
86-
cpu_limit = optional(number)
87-
credits_per_hour = optional(string)
88-
disk_limit = optional(string)
89-
memory_limit = optional(string)
90-
}))
91-
"400cc" = optional(object({
92-
workers = optional(number)
93-
scale = optional(number)
94-
cpu_exclusive = optional(bool)
95-
cpu_limit = optional(number)
96-
credits_per_hour = optional(string)
97-
disk_limit = optional(string)
98-
memory_limit = optional(string)
99-
}))
100-
"600cc" = optional(object({
101-
workers = optional(number)
102-
scale = optional(number)
103-
cpu_exclusive = optional(bool)
104-
cpu_limit = optional(number)
105-
credits_per_hour = optional(string)
106-
disk_limit = optional(string)
107-
memory_limit = optional(string)
108-
}))
109-
"800cc" = optional(object({
110-
workers = optional(number)
111-
scale = optional(number)
112-
cpu_exclusive = optional(bool)
113-
cpu_limit = optional(number)
114-
credits_per_hour = optional(string)
115-
disk_limit = optional(string)
116-
memory_limit = optional(string)
117-
}))
118-
"1200cc" = optional(object({
119-
workers = optional(number)
120-
scale = optional(number)
121-
cpu_exclusive = optional(bool)
122-
cpu_limit = optional(number)
123-
credits_per_hour = optional(string)
124-
disk_limit = optional(string)
125-
memory_limit = optional(string)
126-
}))
127-
"1600cc" = optional(object({
128-
workers = optional(number)
129-
scale = optional(number)
130-
cpu_exclusive = optional(bool)
131-
cpu_limit = optional(number)
132-
credits_per_hour = optional(string)
133-
disk_limit = optional(string)
134-
memory_limit = optional(string)
135-
}))
136-
"3200cc" = optional(object({
137-
workers = optional(number)
138-
scale = optional(number)
139-
cpu_exclusive = optional(bool)
140-
cpu_limit = optional(number)
141-
credits_per_hour = optional(string)
142-
disk_limit = optional(string)
143-
memory_limit = optional(string)
144-
}))
145-
"6400cc" = optional(object({
146-
workers = optional(number)
147-
scale = optional(number)
148-
cpu_exclusive = optional(bool)
149-
cpu_limit = optional(number)
150-
credits_per_hour = optional(string)
151-
disk_limit = optional(string)
152-
memory_limit = optional(string)
153-
}))
154-
}))
36+
sizes = optional(map(object({
37+
workers = optional(number)
38+
scale = optional(number)
39+
cpu_exclusive = optional(bool)
40+
cpu_limit = optional(number)
41+
credits_per_hour = optional(string)
42+
disk_limit = optional(string)
43+
memory_limit = optional(string)
44+
})))
15545
defaultSizes = optional(map(string))
15646
defaultReplicationFactor = optional(map(number))
15747
}))

0 commit comments

Comments
 (0)