Skip to content

Commit a0f693a

Browse files
akoclaude
ako
and
claude
committed
docs: widget BSON version compatibility + CE0463 methodology
Captures the analysis and methodology from closing CE0463 on Mendix 11.9 so future contributors (and future Claude sessions) don't have to re-derive them. Two audiences, two documents: `.claude/skills/debug-bson.md`: - Expands the CE0463 section with the full root-cause spectrum (5 specific triggers seen + mode-dependent visibility) keyed to the commits that fixed each - Documents the "Studio Pro Update Widget" diff methodology — the technique that closed CE0463: snapshot mxcli output, copy project, open in Studio Pro, run "Update widget", snapshot again, diff with UUID normalization. The diff IS the fix. - Adds a note about pairing with `mx dump-mpr` for semantic-level analysis `docs/03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md` (new): - Two-layer model: MPK-derived widget structure (version-resilient) vs Mendix BSON envelope (version-fragile, tied to embedded mendix-11.6 snapshots) - Table of 5 fixes landed for 11.9 compatibility with one-line "why 11.9 cared" per fix - Onboarding-a-new-Mendix-minor procedure (e.g. 11.10) using the same methodology - Map of where envelope-level patches live (embedded JSON, filter widget builder, WidgetValueType serializer, augment helper) — all four must be updated when adding a ubiquitous field - Reference to the unified schema registry proposal as the long-term fix Cross-references: - UNIFIED_SCHEMA_REGISTRY.md "Problem" table now points to the new doc as concrete evidence of the brittleness Phase 4 (codegen from mx dump-mpr) addresses - CLAUDE.md "Useful Files for Context" lists the new doc and updates the debug-bson.md description to mention the diff methodology Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1 parent aea000b commit a0f693a

4 files changed

Lines changed: 217 additions & 7 deletions

File tree

‎.claude/skills/debug-bson.md‎

Lines changed: 35 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -289,11 +289,41 @@ for t, props in crash_props.items():
289289

290290
### CE0463: Widget Definition Changed
291291

292-
**Root cause**: Object property values inconsistent with mode-dependent visibility rules.
293-
294-
**Fix**: Adjust properties based on the widget's current mode. See [PAGE_BSON_SERIALIZATION.md](../../docs/03-development/PAGE_BSON_SERIALIZATION.md#ce0463-widget-definition-changed--root-cause-analysis) for the full analysis.
295-
296-
**Quick workaround**: Run `mx update-widgets` after creating pages.
292+
**Root cause spectrum**: Studio Pro detects the stored widget definition (Type/Object) doesn't match what the runtime expects. Specific triggers seen:
293+
294+
1. **Missing fields on `WidgetValueType`** — e.g. `AllowUpload` added in some 11.x version (commit `ec99cdff`)
295+
2. **WidgetObject.Properties order ≠ WidgetType.PropertyTypes order** — Studio Pro requires identical ordering (commit `b1f4de3a`)
296+
3. **CustomWidget envelope incomplete** — filter widgets missing Appearance/ConditionalEditability/ConditionalVisibility/LabelTemplate (commit `7e6fee84`)
297+
4. **TextTemplate Translation defaults not copied** from PropertyType.ValueType.Translations into WidgetObject's `Template.Items` (commit `f9818394`)
298+
5. **WidgetObject Boolean values diverging from schema default** — `columnsFilterable: 'false'` while schema default is `'true'` (commit `aea000b7`)
299+
6. **Mode-dependent visibility violations** — object property values inconsistent with editor mode rules ([PAGE_BSON_SERIALIZATION.md](../../docs/03-development/PAGE_BSON_SERIALIZATION.md#ce0463-widget-definition-changed--root-cause-analysis))
300+
301+
**Fix methodology (the "Studio Pro Update Widget" diff)** — *the* technique that closed CE0463 on the v0.10 fixture:
302+
303+
1. **Snapshot mxcli output** (the failing state):
304+
```bash
305+
mxcli bson dump page -p test.mpr --object MyMod.MyPage > /tmp/before.json
306+
mxcli bson dump page -p test.mpr --object MyMod.MyPage --format bson > /tmp/before.bson
307+
```
308+
2. **Copy the project** to a path with no `.mpr.lock`, then open in Studio Pro
309+
3. **Right-click the failing widget → "Update widget"** (NOT "Update all widgets" — narrow scope means narrow diff). Save.
310+
4. **Snapshot again**:
311+
```bash
312+
mxcli bson dump page -p studio-pro-copy.mpr --object MyMod.MyPage > /tmp/after.json
313+
```
314+
5. **Diff with UUID normalization** (UUIDs are always different — they'd dominate the diff otherwise):
315+
```python
316+
# normalize: replace {"Subtype": 0, "Data": "..."} with {"Subtype": 0, "Data": "<UUID>"}
317+
# strip $ID fields
318+
# then diff
319+
```
320+
6. **What Studio Pro added/changed/removed = exact fix** mxcli needs
321+
322+
**Pair with `mx dump-mpr` for semantic-level analysis**: `mx dump-mpr --module-names='X' --output-file=dump.json /path/to/project.mpr` gives semantic JSON (typePointer → widget property key resolves automatically). Strip UUIDs and diff to find structural drift the BSON diff would hide behind binary content.
323+
324+
**Investigation methodology used for v0.10 CE0463 fixes** — see [WIDGET_BSON_VERSION_COMPATIBILITY.md](../../docs/03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md) for the full case study and version-resilience model.
325+
326+
**Quick workaround** (if you can't fix the root cause): Run `mx update-widgets` after creating pages.
297327

298328
### CE0642: Property X Is Required
299329

‎CLAUDE.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -521,7 +521,8 @@ Full syntax tables for all MDL statements (microflows, pages, security, navigati
521521
- `reference/mendixmodellib/reflection-data/` - Type definitions with storage names and default values
522522
- `docs/03-development/MDL_PARSER_ARCHITECTURE.md` - ANTLR4 parser design documentation
523523
- `docs/03-development/PAGE_BSON_SERIALIZATION.md` - Page/widget BSON format, type mappings, required defaults
524-
- `.claude/skills/debug-bson.md` - Workflow for debugging BSON serialization issues with `mx` tool
524+
- `docs/03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md` - What's version-resilient vs version-fragile in widget BSON output, and how to onboard a new Mendix minor (e.g. 11.10)
525+
- `.claude/skills/debug-bson.md` - Workflow for debugging BSON serialization issues with `mx` tool (includes the "Studio Pro Update Widget" diff methodology that closed CE0463)
525526
- `cmd/mxcli/lsp.go` - LSP server implementation (hover, definition, diagnostics, completion, symbols)
526527
- `cmd/mxcli/init.go` - `mxcli init` command (project initialization + VS Code extension install)
527528
- `cmd/mxcli/docker/oql.go` - OQL query execution against running Mendix runtime via M2EE admin API

‎docs/03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# Widget BSON Version Compatibility
2+
3+
How mxcli's pluggable-widget BSON output stays in sync (or doesn't) with
4+
specific Mendix versions, and how to extend support to a new minor release.
5+
6+
## Two-layer model
7+
8+
mxcli's widget BSON output is assembled from two sources with very different
9+
version-resilience characteristics:
10+
11+
```
12+
┌────────────────────────────────────────────────────────────────────┐
13+
│ Widget BSON output │
14+
│ │
15+
│ ┌─────────────────────────────────┐ │
16+
│ │ Widget-specific structure │ ← project's .mpk file │
17+
│ │ (PropertyKeys, sub-properties, │ version-tracked per widget │
18+
│ │ attribute types, etc.) │ │
19+
│ └─────────────────────────────────┘ │
20+
│ │
21+
│ ┌─────────────────────────────────┐ │
22+
│ │ Mendix BSON envelope shape │ ← embedded templates │
23+
│ │ (WidgetValueType fields, │ tied to Mendix 11.6 base │
24+
│ │ CustomWidget envelope, │ patched manually for 11.9 │
25+
│ │ array marker conventions) │ │
26+
│ └─────────────────────────────────┘ │
27+
└────────────────────────────────────────────────────────────────────┘
28+
```
29+
30+
**Widget-specific structure** is version-resilient out of the box. `widget init`
31+
parses each project's installed `.mpk` files to derive the per-widget shape;
32+
`sdk/widgets/augment.go` syncs additions and removals between the embedded
33+
template and the installed widget. A new pluggable widget version (e.g.
34+
DataGrid `2.30.1` → `2.31.0`) just works after `mxcli widget init`.
35+
36+
**Mendix BSON envelope shape** is brittle. The embedded templates at
37+
`sdk/widgets/templates/mendix-11.6/*.json` were extracted at Mendix 11.6 and
38+
manually patched as gaps surfaced. Each new Mendix minor that adds a field
39+
to `CustomWidgets$WidgetValueType` or restructures the envelope requires
40+
another round of patching.
41+
42+
## Why this split exists
43+
44+
`.mpk` files declare the widget's own contract (what properties exist, what
45+
types they accept, what defaults the widget author chose). They're shipped
46+
by widget authors and versioned independently of Mendix.
47+
48+
The BSON envelope (`CustomWidgets$WidgetValueType` field set, `WidgetObject`
49+
ordering rules, `Forms$Appearance` structure, etc.) is Mendix runtime infra
50+
that evolves with Mendix itself. There's no per-project file declaring it —
51+
Studio Pro hardcodes the expected shape, and "right" depends on which
52+
Mendix version is reading the BSON.
53+
54+
## Fixes landed for Mendix 11.9 compatibility
55+
56+
| Commit | Field / behavior | Why 11.9 cared |
57+
|---|---|---|
58+
| `ec99cdff` | `AllowUpload: false` on every `WidgetValueType` | Field added in some 11.x; absence triggers CE0463 across every widget |
59+
| `b1f4de3a` | `WidgetObject.Properties` order = `WidgetType.PropertyTypes` order | Studio Pro 11.9 checks for matching ordering; bulk-reordered 5 templates |
60+
| `7e6fee84` | Filter widgets carry full CustomWidget envelope (`Appearance`, `ConditionalEditabilitySettings`, `ConditionalVisibilitySettings`, `LabelTemplate`) | Studio Pro flags incomplete envelopes on nested CustomWidgets |
61+
| `f9818394` | `TextTemplate.Template.Items` populated from `PropertyType.ValueType.Translations` defaults; `Editable: "Always"` on filter widgets | Studio Pro copies translation defaults at widget creation; mxcli left them empty |
62+
| `aea000b7` | `columnsFilterable` and `sortable` Boolean values aligned with their `PropertyType.ValueType.DefaultValue` | Template-extraction bug: stored `false` vs schema-default `true`; Studio Pro detects mismatch |
63+
64+
After these five fixes the v0.10 acceptance fixture
65+
(`mdl-examples/doctype-tests/31-pluggable-datagrid-gallery-v010-examples.mdl`)
66+
emits zero CE0463 errors on a fresh Mendix 11.9 project.
67+
68+
## What's version-stable vs version-fragile
69+
70+
| Element | Source | Version-fragile? |
71+
|---|---|---|
72+
| Widget PropertyKeys (top-level) | MPK XML via `widget init` | ✓ stable |
73+
| Widget property types (Attribute / Expression / TextTemplate / etc.) | MPK XML | ✓ stable |
74+
| Object-list properties (`columns`, `groups`, `series`...) | MPK XML | ✓ stable |
75+
| Sub-property trees inside object lists | MPK XML | ✓ stable |
76+
| Widget version (DataGrid 2.22 → 2.30) | MPK file metadata + augmentation | ✓ stable |
77+
| `CustomWidgets$WidgetValueType` field set | Embedded template + manual patches | ✗ fragile |
78+
| `CustomWidgets$WidgetObject` Properties array ordering | Embedded template | ✗ fragile |
79+
| Required CustomWidget envelope fields | Embedded template + filter widget builder | ✗ fragile |
80+
| TextTemplate default translation population | Embedded template | ✗ fragile |
81+
| Boolean property default consistency | Embedded template | ✗ fragile |
82+
83+
## Onboarding a new Mendix minor (e.g. 11.10, 12.0)
84+
85+
The CE0463 fix methodology used for 11.9 generalizes. Steps:
86+
87+
1. **Download mxbuild for the target version**:
88+
```bash
89+
mxcli setup mxbuild --version 11.10.0
90+
```
91+
92+
2. **Run the v0.10 fixture against a fresh 11.10 project**:
93+
```bash
94+
mxcli new TestApp --version 11.10.0
95+
mxcli widget init -p TestApp/TestApp.mpr
96+
mxcli exec mdl-examples/doctype-tests/31-pluggable-datagrid-gallery-v010-examples.mdl -p TestApp/TestApp.mpr
97+
```
98+
99+
3. **Check with mx**:
100+
```bash
101+
~/.mxcli/mxbuild/11.10.0/modeler/mx check TestApp/TestApp.mpr
102+
```
103+
104+
4. **For each new CE0463** (or other widget validation error):
105+
106+
Use the **"Studio Pro Update Widget" diff** methodology documented in
107+
[`.claude/skills/debug-bson.md`](../../.claude/skills/debug-bson.md#ce0463-widget-definition-changed):
108+
- Snapshot the failing BSON
109+
- Open in Studio Pro 11.10
110+
- "Update widget" on one failing widget instance
111+
- Snapshot again
112+
- Diff (with UUID normalization)
113+
- The diff tells you exactly what to patch in the embedded templates or
114+
the filter widget builder
115+
116+
Each pattern that appears (new envelope field, ordering change, default
117+
value) typically yields a one-line fix and unblocks dozens of widgets.
118+
119+
5. **Add a non-regression test** — see "Cross-version validation" below.
120+
121+
## Where the patches live
122+
123+
- **Embedded templates**: `sdk/widgets/templates/mendix-11.6/*.json` —
124+
for envelope-level fixes that apply to every widget instance loaded from
125+
the embedded template. Most CE0463 fixes land here as bulk-edits across
126+
files (the `AllowUpload` fix added 581 fields across 30 files in one go).
127+
128+
- **Filter widget builder**: `mdl/backend/mpr/datagrid_builder.go`
129+
(`buildFilterWidgetBSON`, `buildMinimalFilterWidgetBSON`, the
130+
`defaultEmptyAppearance` helper) — for the CustomWidget envelope mxcli
131+
constructs around filter widgets inside DataGrid columns.
132+
133+
- **WidgetValueType serializer**: `sdk/mpr/writer_widgets_custom.go`
134+
(`serializeWidgetValueType`) — for the structured-data path (not the
135+
RawType clone path) when building widget BSON from typed inputs.
136+
137+
- **Template augmentation**: `sdk/widgets/augment.go`
138+
(`createDefaultValueType`) — for MPK-derived widget templates when no
139+
embedded template exists.
140+
141+
When patching a field, **update all four paths** if the field is supposed to
142+
be ubiquitous. The CE0463 fixes for `AllowUpload` touched the embedded JSON,
143+
both serializers, and the augment helper.
144+
145+
## Cross-version validation (proposed, not yet implemented)
146+
147+
A `make test-mx-versions` target should:
148+
149+
1. Iterate over a curated list of embedded Mendix versions (LTS + MTS:
150+
10.18, 10.24, 11.6, 11.9, future 11.x as they land)
151+
2. For each: create a blank `.mpr`, run the v0.10 fixture, `mx check` with
152+
that version's `mx` binary
153+
3. Assert zero CE0463 / CE0642 / CE0091 errors
154+
155+
This catches version drift the moment it happens, rather than at user-report
156+
time. Tracked under the unified schema registry effort
157+
([#529](https://github.com/mendixlabs/mxcli/issues/529), Phase 5).
158+
159+
## The long-term answer
160+
161+
The brittleness of the embedded-template layer is exactly what the unified
162+
schema registry proposal addresses
163+
([`docs/11-proposals/UNIFIED_SCHEMA_REGISTRY.md`](../11-proposals/UNIFIED_SCHEMA_REGISTRY.md)).
164+
Phase 4 of that proposal replaces the embedded `mendix-11.6/*.json`
165+
snapshots with templates generated at build time from
166+
`mx dump-mpr` output, parameterized by Mendix version. New Mendix release
167+
support becomes "run codegen against `mx` from that version" rather than
168+
manual patching after CE0463 reports come in.
169+
170+
In the meantime, this doc + the `.claude/skills/debug-bson.md` methodology
171+
keep the patch cadence manageable.
172+
173+
## References
174+
175+
- [`.claude/skills/debug-bson.md`](../../.claude/skills/debug-bson.md) — investigation procedure for CE0463 and related widget BSON errors
176+
- [`docs/03-development/PAGE_BSON_SERIALIZATION.md`](PAGE_BSON_SERIALIZATION.md) — page-level BSON serialization design
177+
- [`docs/03-development/BSON_TOOLING_GUIDE.md`](BSON_TOOLING_GUIDE.md) — `mxcli bson dump` reference
178+
- [Issue #541](https://github.com/mendixlabs/mxcli/issues/541) — the CE0463 case study that motivated this doc
179+
- [Issue #529](https://github.com/mendixlabs/mxcli/issues/529) — unified schema registry proposal (long-term fix)

‎docs/11-proposals/UNIFIED_SCHEMA_REGISTRY.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ Three separate systems, none self-updating, overlapping responsibilities, diverg
2626
|---|---|---|---|
2727
| `reference/mendixmodellib/reflection-data/` | TS SDK extraction | Manual per release | Stale at 11.6, missing storage names / list encodings / ref kinds |
2828
| `supplements.json` | Hand-curated overrides | Per-release human review | Gaps discovered at runtime when Studio Pro rejects output |
29-
| `sdk/widgets/templates/mendix-11.6/` + augmentation | Manual extraction | One-off per Mendix version | Frozen at 11.6; structural BSON shifts not handled |
29+
| `sdk/widgets/templates/mendix-11.6/` + augmentation | Manual extraction | One-off per Mendix version | Frozen at 11.6; structural BSON shifts not handled — see [WIDGET_BSON_VERSION_COMPATIBILITY.md](../03-development/WIDGET_BSON_VERSION_COMPATIBILITY.md) for the five-fix patch sequence that was needed to support 11.9 |
3030
| `.mxcli/widgets/*.def.json` (per-project) | `mxcli widget init` parses MPK XML | User-triggered | Lossy: ignores object-list properties (Accordion `groups`, etc.); no integration with init/refresh |
3131
| `sdk/mpr/system_module.go` | Hand-maintained from 11.6.4 | Manual | Only one version; drifts as System module evolves |
3232

0 commit comments

Comments
 (0)