feat: WITH RESPONSE ENTITIES — schema-driven entity and implicit mapping generation for OpenAPI import

[khode-mx]

Blocked by #268 — implementation should not start until the OpenAPI import PR is merged.

Overview

Document type: Rest$ConsumedRestService (extension of existing OpenAPI import)
Priority: Medium — significantly reduces manual wiring after OpenAPI import
Depends on: CREATE REST CLIENT (OpenAPI:) — implemented in #268

When CREATE REST CLIENT (OpenAPI:) imports a spec, it generates operations with response type JSON but stops short of doing anything with the response schema. In Studio Pro the remaining workflow is: open each operation, execute a test call against the live API, wait for the JSON response, and confirm the entity generation that Studio Pro derives from that payload. The implicit import mapping is then stored inline on the operation.

There is a second, more significant problem with the Studio Pro approach: the live response is treated as the truth. If the API contract defines 20 attributes but a particular test response only returns 5, Studio Pro generates entities and mappings for those 5. The remaining 15 attributes are silently absent. The only recourse is to craft a request that coerces the API into returning a response that includes all attributes — which defeats the purpose, requires intimate knowledge of the API's behaviour that most developers do not have off the top of their head, and is tedious even when that knowledge is available. The resulting domain model is incomplete and the implicit mapping does not reflect what the API can actually return — meaning any microflow built on top of it will silently miss attributes the API is capable of returning, with no indication that anything is absent.

mxcli addresses both problems. The OpenAPI spec schema is the authoritative source — it describes all attributes the API can return, regardless of what any single response happens to include. This proposal adds an optional WITH RESPONSE ENTITIES clause that uses the spec schema directly to generate complete non-persistent entities and populate the Rest$ImplicitMappingResponseHandling inline on each operation, across all operations in one command. No live API access required, and no attributes silently dropped.

The implicit mapping produced here is the same mechanism Studio Pro uses when confirming entity generation from a live response — it is distinct from the standalone ImportMapping document type.

Proposed MDL Syntax

-- Minimal — spec drives everything, no entity generation
CREATE OR MODIFY REST CLIENT PetStore.PetStoreAPI (
  OpenAPI: 'specs/petstore.json'
);

-- With response entity generation
CREATE OR MODIFY REST CLIENT PetStore.PetStoreAPI (
  OpenAPI: 'specs/petstore.json'
)
WITH RESPONSE ENTITIES;

-- Combined with BaseUrl override
CREATE OR MODIFY REST CLIENT PetStore.PetStoreAPI (
  OpenAPI: 'specs/petstore.json',
  BaseUrl: 'https://staging.petstore.example.com/v3'
)
WITH RESPONSE ENTITIES;

WITH RESPONSE ENTITIES is a clause on the statement, not a property inside the parentheses. This follows the structural pattern of CREATE EXTERNAL ENTITIES FROM Module.ODataService — entity generation is expressed as part of what the statement does, not as a boolean flag. Both WITH and ENTITIES are existing lexer tokens; no new tokens are required.

The clause is only valid when OpenAPI: is present. Using it without OpenAPI: is a parse-time error.

What Gets Generated

For each operation whose 200 response (or first 2xx response) contains content["application/json"].schema:

1. A non-persistent entity per object schema, placed in the module's domain model
2. Scalar attributes for each primitive property in the schema
3. Reference associations for nested object properties; ReferenceSet associations for array items
4. A Rest$ImplicitMappingResponseHandling record on the operation, with a full ImportMappings$ObjectMappingElement / ImportMappings$ValueMappingElement tree matching the entity structure

Example

Given a Petstore operation GET /pet/{petId} with response schema referencing #/components/schemas/Pet (id: int64, name: string, status: string), mxcli generates:

CREATE NON-PERSISTENT ENTITY PetStore.Pet (
  id:     Long,
  name:   String(0),
  status: String(0)
);

And wires the implicit mapping on the operation equivalent to:

response: mapping PetStore.Pet {
  'id'     = id,
  'name'   = name,
  'status' = status
}

Entity Naming Convention

Schema source	Entity name
Named $ref component (#/components/schemas/Pet)	Component name verbatim → Module.Pet
Inline schema on operation	{OperationName}Response → Module.GetPetByIdResponse
Nested object property	{ParentEntityName}_{PropertyName} → Module.Pet_Tags
Array items object	{ParentEntityName}_{PropertyName}Item → Module.Order_ItemsItem

If an entity with the derived name already exists in the module, it is reused rather than recreated. Its attributes are not modified — the user is responsible for keeping existing entities aligned with the schema.

Schema Traversal Rules

In scope

• $ref resolution from #/components/schemas/ (local references only)
• Nested object schemas (recursive, cycle-safe)
• Array items that are object schemas → ReferenceSet + child entity
• Scalar types: string, integer (int32 → Integer, int64 → Long), number (→ Decimal), boolean, string/date-time (→ DateTime), string/date (→ Date)
• nullable: true — attribute is created; nullability is implicit in Mendix

Warnings emitted, element skipped

• allOf / oneOf / anyOf — warn, treat containing property as String(0)
• Array items that are scalar (not object) — warn, treat as String(0) attribute
• $ref pointing outside #/components/schemas/ (e.g. external file refs) — warn, skip
• Operation has no response schema defined (absent responses block, or response has no content) — warn, skip entity generation for that operation; operation itself is still created with Rest$NoResponseHandling

Cycle detection

If a $ref chain produces a cycle (e.g. Pet → Category → Pet), mxcli detects it, breaks the cycle at the second occurrence, and emits a warning naming the cycle path.

Response selection

Use responses["200"] when present. Fall back to the first 2xx response key found. If no 2xx response exists or none has application/json content, the operation gets Rest$NoResponseHandling as before — no entity is generated for that operation.

BSON Structure

No new BSON types are introduced. The generated output uses existing types already supported by the parser and writer:

BSON $Type	Purpose
Rest$ImplicitMappingResponseHandling	Response handling on a CRS operation — already implemented
ImportMappings$ObjectMappingElement	Object-level mapping node — already implemented
ImportMappings$ValueMappingElement	Scalar attribute mapping — already implemented
DomainModels$Entity (non-persistent)	Entity in domain model — already implemented
DomainModels$Attribute	Attribute on entity — already implemented
DomainModels$Association	Reference / ReferenceSet association — already implemented

Implementation Plan

Phase 1 — AST: add WithResponseEntities bool to CreateRestClientStmt in mdl/ast/ast_rest.go

Phase 2 — Grammar: add (WITH RESPONSE ENTITIES)? clause to createRestClientStatement in mdl/grammar/MDLParser.g4; run make grammar

Phase 3 — Visitor: detect WITH RESPONSE ENTITIES context in ExitCreateRestClientStatement(), set the flag

Phase 4 — New file mdl/openapi/schema_to_entities.go: BuildResponseEntities(spec, moduleName) with recursive buildEntity(), cycle detection, deterministic map iteration

Phase 5 — Executor: after service is built in createRestClientFromSpec(), if flag is set, call BuildResponseEntities, write entities + associations via ctx.Backend, patch implicit mappings onto operations

Phase 6 — Skill: extend Approach 0 in cmd/mxcli/skills/rest-client.md with WITH RESPONSE ENTITIES example and what gets generated

Phase 7 — MDL examples: 21-openapi-response-entities-examples.mdl covering $ref schemas, inline schemas, nested objects, arrays, allOf warning

Version Requirements

No new version gate — covered by existing rest_client_basic (Mendix 10.1+).

Validation Checklist (before PR)

1. make grammar passes after grammar change
2. mxcli check accepts the new clause syntax
3. CREATE OR MODIFY REST CLIENT ... WITH RESPONSE ENTITIES against the Petstore spec: correct entity count, attributes match schema, associations present
4. DESCRIBE REST CLIENT roundtrip — implicit mappings visible in output
5. Studio Pro opens the resulting .mpr, operations show the implicit mapping correctly
6. Warning emitted for allOf/oneOf schemas; operation still created without mapping

Out of Scope / Follow-up

The items below were considered during design and deliberately excluded to keep this proposal bounded to a single, shippable concern.

• Request body → export entities: The same schema-driven generation pattern applies in the other direction — request body schemas could generate non-persistent parameter entities and wire Rest$ImplicitMappingBody. Excluded because the response direction validates the core traversal and BSON wiring approach first. Once shipped and confirmed in Studio Pro, the export direction is a low-risk follow-up with a near-identical implementation path.

• ALTER REST CLIENT ... REFRESH with entity reconciliation: Worth noting, but this is a follow-up to the base OpenAPI import feature, not to this proposal. The Consumed REST Service document currently has no update path at all — once created from a spec, the contract is frozen. The OpenAPI spec serves as input only, with no mechanism to re-apply a newer version to an existing document. That limitation exists independently of whether WITH RESPONSE ENTITIES was used. Any refresh capability requires first solving the broader problem of updating an existing CRS from a new spec, which is a substantial standalone effort covering both REST and OData and is tracked in a separate contract refresh proposal.

• Standalone CREATE RESPONSE ENTITIES FROM Module.Service: Post-import entity generation for an already-imported service. Deferred because the OR MODIFY pattern on CREATE REST CLIENT already covers the re-run case cleanly.

Related

• feat: import consumed REST client from OpenAPI 3.0 spec #268 — OpenAPI import PR this extends (must merge first)
• mdl/openapi/parser.go — OpenAPI spec parser to extend
• mdl/executor/cmd_rest_clients.go — executor to extend
• mdl/ast/ast_rest.go — AST to extend
• mdl/visitor/visitor_rest.go — visitor to extend
• mdl/grammar/MDLParser.g4:2494 — createRestClientStatement rule to extend
• sdk/mpr/writer_rest.go — existing Rest$ImplicitMappingResponseHandling writer (no changes needed)

[github-actions]

Issue Triage

Summary:
The issue proposes adding a WITH RESPONSE ENTITIES clause to the CREATE OR MODIFY REST CLIENT statement so that mxcli can generate non‑persistent entities and implicit response mappings directly from an OpenAPI schema, eliminating the need for manual test calls and preventing missing attributes.

Checklist for feature request completeness

Required information	Present?	Notes
Use case – why the feature is needed, what problem it solves	✅	Detailed explanation of the current manual workflow and its shortcomings.
Proposed syntax or behavior (if applicable)	✅	Full MDL syntax examples, entity naming rules, schema traversal, warnings, and implementation plan.
Mendix version relevance (if version‑specific)	✅	States that no new version gate is needed; relies on existing rest_client_basic (Mendix 10.1+).

Conclusion:
All required information for a feature request is provided. Thank you for the thorough proposal!

---

Automated triage via OpenRouter — workflow source