Skip to content

Commit c95d9bf

Browse files
akoclaude
andcommitted
docs: update ARCHITECTURE.md with backend abstraction and pluggable widget engine
- Add Backend Layer to high-level architecture diagram (FullBackend, PageMutator, WorkflowMutator, BackendFactory, MockBackend) - Update executor arrow to point through Backend interfaces, not SDK directly - Add mdl/backend/, mdl/types/, mdl/bsonutil/ to package structure - Update MDL Layer package table with new packages and accurate descriptions - Add design decision #6: Pluggable Widget Engine (WidgetRegistry 3-tier loading, OperationRegistry, .def.json format, mxcli widget extract, CE0463 augmentation pipeline) — documents engalar's PRs #28/#68/#166 - Add design decision #11: Backend Abstraction + Dependency Inversion (executor/backend separation rule, mutation factory pattern, mdl/types convention) — documents retran's refactoring stack #225/#229/#235-#239 - Renumber design decisions 6-9 → 7-10 to accommodate new sections Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
1 parent 009eb28 commit c95d9bf

1 file changed

Lines changed: 173 additions & 6 deletions

File tree

docs/01-project/ARCHITECTURE.md

Lines changed: 173 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,20 @@ graph TB
4949
SQLGEN[Connector Generator]
5050
end
5151
52+
subgraph "Backend Layer (mdl/backend/)"
53+
BACKEND[FullBackend interface]
54+
MPRBACK[MPR Backend impl]
55+
MOCKBACK[Mock Backend]
56+
PAGEMUT[PageMutator]
57+
WFMUT[WorkflowMutator]
58+
FACTORY[BackendFactory]
59+
end
60+
61+
subgraph "Shared Types (mdl/types/)"
62+
TYPES[Domain types]
63+
BSONUTIL[bsonutil/]
64+
end
65+
5266
subgraph "Storage Layer"
5367
READER[MPR Reader]
5468
WRITER[MPR Writer]
@@ -76,11 +90,17 @@ graph TB
7690
PARSER --> AST
7791
AST --> VISITOR
7892
VISITOR --> AST
79-
EXEC --> SDK
93+
EXEC --> BACKEND
8094
EXEC --> SQLPKG
8195
EXEC --> CATALOG
8296
EXEC --> LINTER
8397
98+
BACKEND --> PAGEMUT
99+
BACKEND --> WFMUT
100+
MPRBACK --> BACKEND
101+
MOCKBACK --> BACKEND
102+
FACTORY --> MPRBACK
103+
84104
SDK --> DM
85105
SDK --> MF
86106
SDK --> PG
@@ -89,6 +109,11 @@ graph TB
89109
SDK --> READER
90110
SDK --> WRITER
91111
112+
MPRBACK --> READER
113+
MPRBACK --> WRITER
114+
MPRBACK --> TYPES
115+
MPRBACK --> BSONUTIL
116+
92117
SQLPKG --> SQLCONN
93118
SQLPKG --> SQLQUERY
94119
SQLPKG --> SQLIMPORT
@@ -120,6 +145,11 @@ graph LR
120145
CATALOGPKG[catalog/]
121146
LINTERPKG[linter/]
122147
REPLPKG[repl/]
148+
BACKENDPKG[backend/]
149+
BACKENDMPR[backend/mpr/]
150+
BACKENDMOCK[backend/mock/]
151+
TYPESPKG[types/]
152+
BSONUTILPKG[bsonutil/]
123153
end
124154
125155
subgraph "api/"
@@ -236,7 +266,12 @@ sequenceDiagram
236266
| `mdl/grammar` | ANTLR4 lexer/parser (generated from MDLLexer.g4 + MDLParser.g4) |
237267
| `mdl/ast` | AST node types for MDL statements |
238268
| `mdl/visitor` | ANTLR listener that builds AST from parse tree |
239-
| `mdl/executor` | Executes AST nodes against the SDK (~45k lines across 40+ files); handles domain model, microflows, pages, security, navigation, SQL, imports, OData, and more |
269+
| `mdl/executor` | Thin orchestrator: parses AST, calls `ctx.Backend.*`, formats output. **No `sdk/mpr` imports.** |
270+
| `mdl/backend` | Domain-specific backend interfaces (`FullBackend`, `PageMutator`, `WorkflowMutator`, `BackendFactory`) |
271+
| `mdl/backend/mpr` | MPR-backed implementation of all backend interfaces; owns all BSON mutation logic |
272+
| `mdl/backend/mock` | `MockBackend` with Func-field injection for unit testing without a `.mpr` file |
273+
| `mdl/types` | Shared domain types (`NavigationDocument`, `JavaAction`, `JsonStructure`, EDMX/AsyncAPI parsers, ID utilities) — no `sdk/mpr` dependency |
274+
| `mdl/bsonutil` | CGO-free BSON ID utilities (`IDToBsonBinary`, `BsonBinaryToID`, `NewIDBsonBinary`) |
240275
| `mdl/catalog` | SQLite-based catalog for querying project metadata (entities, microflows, references, permissions, source code) |
241276
| `mdl/linter` | Extensible linting framework with built-in rules and Starlark scripting support; includes report generation |
242277
| `mdl/repl` | Interactive REPL interface |
@@ -720,7 +755,85 @@ sdk/widgets/templates/mendix-11.6/
720755
- Expression-type properties require non-empty values (template may have placeholders)
721756
- See [PAGE_BSON_SERIALIZATION.md](../03-development/PAGE_BSON_SERIALIZATION.md) for detailed serialization rules
722757

723-
### 6. Catalog System
758+
### 6. Pluggable Widget Engine
759+
760+
The **PluggableWidgetEngine** is a data-driven system that replaces hardcoded widget builders with a registry of `.def.json` widget definition files. It handles CREATE, INSERT, and ALTER operations for all pluggable (React client-side) widgets.
761+
762+
```mermaid
763+
flowchart TD
764+
subgraph "WidgetRegistry (3-tier)"
765+
EMBEDDED["Embedded definitions\n(sdk/widgets/definitions/)"]
766+
GLOBAL["Global definitions\n(~/.mxcli/widgets/)"]
767+
PROJECT["Project definitions\n(.mxcli/widgets/)"]
768+
end
769+
770+
subgraph "PluggableWidgetEngine"
771+
DEF[".def.json definition"]
772+
OPS["OperationRegistry\n(attribute, association,\nprimitive, selection,\ndatasource, widgets)"]
773+
COND["Condition evaluator"]
774+
MAPPER["Property mapper"]
775+
end
776+
777+
subgraph "Output"
778+
TMPL["Load + clone template\n(sdk/widgets/templates/)"]
779+
AUG["Augment from .mpk\n(sdk/widgets/augment.go)"]
780+
BSON["Serialize to BSON"]
781+
end
782+
783+
PROJECT -->|overrides| GLOBAL
784+
GLOBAL -->|overrides| EMBEDDED
785+
EMBEDDED --> DEF
786+
DEF --> OPS
787+
OPS --> COND
788+
COND --> MAPPER
789+
MAPPER --> TMPL
790+
TMPL --> AUG
791+
AUG --> BSON
792+
```
793+
794+
**Widget definition format (`.def.json`):**
795+
Each file describes a single widget type — its MDL property mappings, conditions, and operation types:
796+
```json
797+
{
798+
"widgetId": "com.mendix.widget.web.combobox.ComboBox",
799+
"properties": [
800+
{ "key": "attribute", "op": "attribute", "mdl": "ATTRIBUTE" },
801+
{ "key": "datasource", "op": "datasource", "mdl": "DATA_SOURCE" }
802+
]
803+
}
804+
```
805+
806+
**WidgetRegistry — 3-tier loading:**
807+
1. Embedded definitions (compiled into the binary via `go:embed`)
808+
2. Global user definitions (`~/.mxcli/widgets/`) — override embedded
809+
3. Project-level definitions (`.mxcli/widgets/`) — override global
810+
811+
This allows users to add support for custom or third-party widgets without recompiling mxcli.
812+
813+
**OperationRegistry — 6 built-in operation types:**
814+
815+
| Operation | Description |
816+
|-----------|-------------|
817+
| `attribute` | Bind an entity attribute |
818+
| `association` | Bind an entity association |
819+
| `primitive` | Set a primitive value (string, bool, int) |
820+
| `selection` | Set a selection enum value |
821+
| `datasource` | Configure a data source (Database, Microflow, Nanoflow) |
822+
| `widgets` | Nest child widgets in a container slot |
823+
824+
**`mxcli widget extract` CLI command:**
825+
Generates a skeleton `.def.json` from a `.mpk` widget package file, making it easy to add support for new third-party widgets:
826+
```bash
827+
mxcli widget extract --mpk path/to/widget.mpk --output .mxcli/widgets/
828+
```
829+
830+
**Augmentation pipeline (CE0463 prevention):**
831+
When a widget is written to an MPR, `AugmentTemplate` reconciles the embedded template against the project's installed `.mpk` version — adding missing properties and removing stale ones, including **nested ObjectType properties** (e.g., DataGrid2 column sub-properties). This prevents the CE0463 "widget definition changed" error when the project's widget version differs from the embedded template.
832+
833+
Key files: `mdl/executor/widget_engine.go` (engine), `mdl/executor/widget_registry.go` (registry), `sdk/widgets/augment.go` (augmentation), `sdk/widgets/mpk/mpk.go` (`.mpk` parser).
834+
835+
### 7. Catalog System
836+
724837

725838
The SQLite-based catalog (`mdl/catalog/`) enables cross-reference queries and code search:
726839

@@ -740,7 +853,7 @@ flowchart LR
740853

741854
Builders populate tables for modules, entities, microflows, pages, permissions, references, and source code.
742855

743-
### 7. Credential Isolation for External SQL
856+
### 8. Credential Isolation for External SQL
744857

745858
External database credentials are managed through environment variables or YAML config, never stored in MDL scripts:
746859

@@ -751,11 +864,11 @@ DSN resolution order:
751864
3. Inline connection string (development only)
752865
```
753866

754-
### 8. Pure Go / No CGO
867+
### 9. Pure Go / No CGO
755868

756869
The project uses `modernc.org/sqlite` (a pure Go SQLite implementation) to eliminate the CGO dependency. This simplifies cross-compilation and deployment — no C compiler is required.
757870

758-
### 9. Multi-Version Support
871+
### 10. Multi-Version Support
759872

760873
Mendix projects vary along three versioning axes: **platform version** (9.x–11.x), **widget version** (each project bundles specific `.mpk` widget packages), and **extension documents** (Mendix 11+ custom document types). The BSON document structure changes across all three.
761874

@@ -778,6 +891,60 @@ Key files: `sdk/widgets/augment.go` (augmentation logic), `sdk/widgets/mpk/mpk.g
778891

779892
**Planned: Schema Registry** (`sdk/schema/`): A runtime registry loaded from reflection data that provides type metadata (storage names, defaults, reference kinds, list encodings) per Mendix version. This will complement the hand-coded parsers/writers by handling field completeness, validation, and version migration. See [Multi-Version Support](../11-proposals/MULTI_VERSION_SUPPORT.md) for the full architecture and implementation status.
780893

894+
### 11. Backend Abstraction + Dependency Inversion
895+
896+
The executor **never imports `sdk/mpr`**. All project access goes through `ctx.Backend`, which implements `backend.FullBackend`. This enables:
897+
- Unit tests without a `.mpr` file (inject `MockBackend`)
898+
- Alternative storage backends (cloud, in-memory, etc.)
899+
- Isolated BSON mutation logic in `mdl/backend/mpr/`
900+
901+
```mermaid
902+
flowchart LR
903+
subgraph "Executor (thin orchestrator)"
904+
HANDLER["applySetProperty(ctx, op)"]
905+
CTX["ctx.Backend.OpenPageForMutation(id)"]
906+
end
907+
908+
subgraph "backend.PageMutator interface"
909+
SET["SetWidgetProperty(ref, prop, value)"]
910+
INSERT["InsertWidget(target, pos, widgets)"]
911+
SAVE["Save()"]
912+
end
913+
914+
subgraph "mdl/backend/mpr (BSON impl)"
915+
BSON["Walk BSON tree, mutate, marshal"]
916+
end
917+
918+
subgraph "mdl/backend/mock (test impl)"
919+
FUNC["SetWidgetPropertyFunc(ref, prop, value)"]
920+
end
921+
922+
HANDLER --> CTX
923+
CTX --> SET
924+
SET --> BSON
925+
SET --> FUNC
926+
```
927+
928+
**Rule for new executor commands:**
929+
1. Add a method to the appropriate domain interface in `mdl/backend/` (e.g., `DomainModelBackend`, `MicroflowBackend`)
930+
2. Implement it in `mdl/backend/mpr/` operating on BSON/reader/writer
931+
3. Add a `Func`-field stub in `mdl/backend/mock/`
932+
4. Call `ctx.Backend.YourMethod()` from the executor handler
933+
5. Never call `sdk/mpr` types directly from the executor
934+
935+
**Mutation pattern (ALTER PAGE / ALTER WORKFLOW):**
936+
937+
For operations that modify a document in place, use the `PageMutator`/`WorkflowMutator` factory pattern:
938+
1. `mutator, err := ctx.Backend.OpenPageForMutation(unitID)` — opens the document
939+
2. Call mutator methods (`SetWidgetProperty`, `InsertWidget`, etc.)
940+
3. `mutator.Save()` — persists the changes
941+
942+
The mutator owns the document's lifecycle; the executor only describes *what* to change.
943+
944+
**Shared types (`mdl/types/`):**
945+
946+
Types used by both `mdl/` and `sdk/mpr` live in `mdl/types/`. The `sdk/mpr` package re-exports them as type aliases (`type JavaAction = types.JavaAction`) for backward compatibility. New shared types go in `mdl/types/`, not in `sdk/mpr/reader_types.go`.
947+
781948
## Future Architecture Considerations
782949

783950
1. **48 of 52 Metamodel Domains**: Workflows, REST services, and many other domains are not yet implemented

0 commit comments

Comments
 (0)