fix(gateway): preserve plugin data-list fields; harden DTO schema and dedup handler helpers

GET /{entity}/data now returns the opaque DataListResult envelope (mirroring
the fault list route) instead of the typed Collection<DataItem>. The plugin
branch previously re-parsed the provider payload through a lenient
JsonReader<Collection<DataItem>>, which silently dropped any per-item field
outside {id,name,category} - including the OPC-UA plugin's value/unit/data_type/
writable. It now passes the provider payload through verbatim. Runtime entities
keep a byte-identical {items, x-medkit} wire shape, built from the typed
Collection<DataItem, DataListXMedkit> and wrapped into the envelope.

Also:
- schema_writer: optional enum fields attach `enum` to the non-null anyOf
  branch instead of as a top-level sibling, so the nullable claim and the enum
  constraint no longer contradict each other.
- Add a compile-time guard (static_assert on a non-empty dto_name<T>) in the
  schema registry and in schema_of, so a missing dto_name fails the build
  instead of producing an empty schema key and a dangling $ref.
- Hoist the duplicated make_error and flatten_validator_error helpers (copied
  across 12 handler translation units) into a shared handler_support.hpp.

Tests:
- Restore the sorted_contributors local-first/alphabetical ordering regression
  test that was lost with the removed x-medkit suite.
- Add JsonReader enum-rejection and optional-enum schema tests.
- Add a DataListResult regression test asserting the opaque envelope preserves
  vendor per-item fields and that the typed re-parse would have dropped them.

Docs: document the opaque data-list schema, the optional fields that change
from explicit null to omitted, and the always-present entity type discriminator.

Author: bburda

src/ros2_medkit_gateway/CHANGELOG.rst

@@ -44,9 +44,16 @@ Unreleased
   gateway still omits absent optional fields, and ``JsonReader`` continues
   to accept absent fields; the schema change only opts the published spec
   into round-tripping a literal ``null`` value cleanly for clients that
-  prefer to send one. The ``rtmaps_medkit`` variant is explicitly NOT
-  covered by this PR - its handlers continue to run on the pre-typed
-  HandlerContext surface and will be migrated separately
+  prefer to send one. As part of this, a handful of fields that were
+  previously emitted as an explicit JSON ``null`` when absent are now omitted
+  entirely (consistent with the optional-omission policy): the script
+  execution fields ``progress`` / ``started_at`` / ``completed_at`` /
+  ``parameters`` / ``error`` (``GET .../scripts/{id}/executions/{eid}``) and
+  the script ``parameters_schema`` field (``GET .../scripts/{id}``). Clients
+  that tested ``field === null`` or relied on the key always being present must
+  treat an absent key the same as ``null``. The ``rtmaps_medkit`` variant is
+  explicitly NOT covered by this PR - its handlers continue to run on the
+  pre-typed HandlerContext surface and will be migrated separately
   (`#403 <https://github.com/selfpatch/ros2_medkit/issues/403>`_)
 * Synchronous operation-execution service-call failures
   (``POST /api/v1/{entity-path}/operations/{id}/executions`` when the underlying
@@ -59,14 +66,34 @@ Unreleased
   parsed ``error.code`` / ``error.details`` for this specific failure must read
   ``vendor_code`` / ``parameters`` instead
   (`#403 <https://github.com/selfpatch/ros2_medkit/issues/403>`_)
+* ``GET /api/v1/{entity-path}/data`` now publishes the opaque ``DataListResult``
+  schema (``{type: object, additionalProperties: true, x-medkit-opaque: true}``),
+  matching how ``GET .../faults`` already publishes ``FaultListResult``. The wire
+  payload is unchanged for runtime (ROS 2) entities - it is still
+  ``{"items": [...], "x-medkit": {...}}`` built from the typed
+  ``Collection<DataItem, DataListXMedkit>`` - but for plugin-owned entities the
+  provider's free-form per-item shape now passes through verbatim instead of
+  being re-parsed into ``Collection<DataItem>``. This fixes a regression in which
+  plugin per-item fields (the OPC-UA plugin's ``value`` / ``unit`` / ``data_type``
+  / ``writable``) were silently dropped by the typed re-parse. Clients that
+  generated a typed ``DataItem`` model from the previous spec for this route now
+  see an opaque object instead
+  (`#403 <https://github.com/selfpatch/ros2_medkit/issues/403>`_)
+* Entity responses (areas, components, apps, functions - list items and detail)
+  now always carry a top-level ``type`` discriminator (an enum of
+  ``area`` / ``component`` / ``app`` / ``function``). Previously list items had no
+  ``type`` and detail responses exposed it only inside ``x-medkit.entityType``.
+  Additive and tolerant-client-safe; consumers keying on the entity kind can now
+  read the top-level ``type``
+  (`#403 <https://github.com/selfpatch/ros2_medkit/issues/403>`_)
 * ``ros2_medkit_msgs/srv/ClearFault`` request gains a ``bool skip_correlation_auto_clear`` field (see the per-entity fault scope entry below for the in-tree motivation). Adding a request field changes the service type hash, so out-of-tree callers that invoke the service directly (for example ``ros2 service call /fault_manager/clear_fault ros2_medkit_msgs/srv/ClearFault ...`` as documented in the ``ros2_medkit_fault_manager`` README) must rebuild against the new ``ros2_medkit_msgs`` to keep talking to ``fault_manager``. The in-tree gateway client and server are updated together (`#395 <https://github.com/selfpatch/ros2_medkit/issues/395>`_)
 * Per-entity fault routes are now correctly scoped to the entity's hosted apps. ``GET /api/v1/{entity-path}/faults/{fault_code}``, ``DELETE /api/v1/{entity-path}/faults/{fault_code}``, ``GET /api/v1/{entity-path}/faults``, and ``DELETE /api/v1/{entity-path}/faults`` previously fell back to a prefix match against the entity's ``namespace_path``; when that was empty (host-derived / synthetic components, manifest components without a ``namespace`` field, Areas, Functions, and Apps with a wildcard ``ros_binding.namespace_pattern``) the scope filter was silently disabled and the routes exposed - and on ``DELETE``, cleared - faults reported by apps that belonged to entirely different entities. All four handlers now resolve the addressed entity to its hosted-app FQN set (via the new ``HandlerContext::resolve_entity_source_fqns`` helper) and apply a strict all-sources scope check: a fault counts as in scope only when **every** entry in its ``reporting_sources`` is owned by the entity (exact FQN match, or strict path-child via ``<fqn>/...``). Per-fault routes return ``404 Resource Not Found`` for any fault that fails the check; collection routes return an empty ``items`` array. The underlying ``GetFault.srv`` contract is unchanged; ``ClearFault.srv`` gains a new ``skip_correlation_auto_clear`` request flag so per-entity DELETE can opt out of cascade-clearing correlated symptom fault codes that may live in other entities. Per-entity collection responses no longer include the global ``muted_count`` / ``cluster_count`` / ``muted_faults`` / ``clusters`` correlation metadata; those remain on the global ``GET /api/v1/faults`` route. Behavior changes visible to clients: (a) faults reported by apps outside the addressed entity are no longer returned or cleared via that entity's route, (b) **mixed-source** faults that include at least one out-of-entity reporter are likewise rejected with ``404`` on per-fault routes and excluded from per-entity collection responses (use the global ``GET /api/v1/faults`` to see them), (c) per-entity DELETE no longer cascade-clears correlated symptoms outside the entity (`#395 <https://github.com/selfpatch/ros2_medkit/issues/395>`_)
 * ``GET /api/v1/updates/{id}/status`` no longer returns ``404`` for a registered-but-idle package; ``POST /api/v1/updates`` now seeds a ``pending`` status, so the endpoint returns ``200 {"status": "pending"}`` immediately after registration. ``404`` is reserved for packages that are not registered. Clients that used ``404`` as a signal for "registered but nothing started yet" must adapt (`#378 <https://github.com/selfpatch/ros2_medkit/issues/378>`_)
 
 **Features:**
 
 * Typed ``fan_out_collection<T>`` aggregating helper replaces raw-JSON ``merge_peer_items`` on the typed collection routes (data, operations, config, logs). Peer items are decoded via ``dto::JsonReader<T>``; items that fail validation are removed from the merged ``items`` array, recorded in ``x-medkit.peer_dropped_items`` with the JsonReader error plus a best-effort ``source_id``, and logged at ``WARN``. Items that parse successfully are re-serialized through the local ``dto::JsonWriter<T>``, so any peer-supplied fields outside the local DTO schema are dropped from the merged response (the previous raw passthrough preserved unknown peer fields verbatim). Previously, malformed peer items silently disappeared into the merged response; fleet operators can now detect inter-gateway schema drift directly on the wire (`#403 <https://github.com/selfpatch/ros2_medkit/issues/403>`_)
-* ``Collection<T, XMedkitT>`` is now a 2-parameter template. Domain list endpoints (faults, config, data, logs) reference their richer per-domain collection x-medkit struct (``FaultListXMedkit``, ``ConfigListXMedkit``, ``DataListXMedkit``, ``LogListXMedkit``) directly in the published schema instead of the generic ``XMedkitCollection``, so generated clients see aggregation counts, peer provenance, and ``peer_dropped_items`` from the schema (`#403 <https://github.com/selfpatch/ros2_medkit/issues/403>`_)
+* ``Collection<T, XMedkitT>`` is now a 2-parameter template. Domain list endpoints (faults, config, logs) reference their richer per-domain collection x-medkit struct (``FaultListXMedkit``, ``ConfigListXMedkit``, ``LogListXMedkit``) directly in the published schema instead of the generic ``XMedkitCollection``, so generated clients see aggregation counts, peer provenance, and ``peer_dropped_items`` from the schema. The data list builds the same typed ``Collection<DataItem, DataListXMedkit>`` internally (so the wire still carries those fields) but publishes the opaque ``DataListResult`` envelope, because plugin-owned data entities can return vendor per-item fields the typed item schema cannot describe (see the data-list breaking-change entry above) (`#403 <https://github.com/selfpatch/ros2_medkit/issues/403>`_)
 * New ``opaque_object("key", &T::field)`` DTO field descriptor in ``dto/contract.hpp``. Binds a ``nlohmann::json`` member as a typed "any JSON object" field: ``JsonWriter`` emits it verbatim, ``JsonReader`` rejects scalars / arrays / null, ``SchemaWriter`` emits ``{type: object, additionalProperties: true, x-medkit-opaque: true}``. Used for fields whose runtime shape is decided by an upstream component the gateway cannot introspect (live ROS message payloads, plugin-defined fault envelopes, action results) (`#403 <https://github.com/selfpatch/ros2_medkit/issues/403>`_)
 * ``GET /api/v1/faults/stream`` event payloads now carry an optional ``x-medkit`` SOVD payload-extension object with ``entity_type`` and ``entity_id`` fields. When the gateway can resolve the fault's first reporting source back to a SOVD entity (via the manifest-mode linking index, or a runtime-mode last-segment match against an existing App), consumers can hit ``/{entity_type}/{entity_id}/bulk-data/rosbags/{fault_code}`` directly instead of HEAD-probing every entity. Resolution is snapshotted at event arrival, so a discovery refresh between enqueue and stream-out cannot retroactively change the entity reported to consumers. The ``x-medkit`` object is omitted entirely when no entity can be resolved, so existing SSE consumers ignore the addition (`#380 <https://github.com/selfpatch/ros2_medkit/issues/380>`_)
 * Plugin API version bumped to v7. Adds ``PluginContext::notify_entities_changed(EntityChangeScope)`` lifecycle hook for plugins that mutate the entity surface at runtime; default no-op keeps v6 source code compiling unchanged against v7 headers. Binary compatibility is not provided: the plugin loader uses a strict equality check on ``plugin_api_version()``, so out-of-tree plugins must be recompiled (`#376 <https://github.com/selfpatch/ros2_medkit/issues/376>`_)

src/ros2_medkit_gateway/design/dto_contract.rst

@@ -690,10 +690,21 @@ endpoints (areas, components, apps, functions) use the default
 ``XMedkitCollection`` x-medkit; the domain collection endpoints specialise
 ``XMedkitT`` to their richer per-domain shape (``FaultListXMedkit``,
 ``FaultListAggXMedkit``, ``ConfigListXMedkit``, ``DataListXMedkit``,
-``LogListXMedkit``), so the published schema for each list response now
-references the actual collection x-medkit struct. Generated clients see the
-exact aggregation, peer-provenance, and ``peer_dropped_items`` fields that
-appear on the wire.
+``LogListXMedkit``). For the config and log list routes the published schema
+references the actual collection x-medkit struct directly, so generated clients
+see the exact aggregation, peer-provenance, and ``peer_dropped_items`` fields
+that appear on the wire.
+
+The fault and data list routes are the exception: they publish the opaque
+``FaultListResult`` / ``DataListResult`` envelopes rather than the typed
+``Collection<...>`` schema, because plugin-owned entities can return
+vendor-specific per-item shapes that the typed item schema cannot describe.
+The data list handler still *builds* a typed
+``Collection<DataItem, DataListXMedkit>`` for runtime (ROS 2) entities and
+serializes it into the envelope (so the wire shape - including
+``peer_dropped_items`` - is unchanged), but the plugin branch passes the
+provider's free-form payload through verbatim. See "Opaque Object Policy" and
+the "Provider ABI" section above.
 
 Key Files
 ---------

src/ros2_medkit_gateway/include/ros2_medkit_gateway/core/http/handlers/data_handlers.hpp

@@ -52,10 +52,16 @@ class DataHandlers {
    *
    * GET /{entities}/{id}/data
    *
-   * Returns SOVD ValueMetadata items (typed `DataItem`) with x-medkit ROS2-
-   * specific extensions plus typed fan-out via `fan_out_collection<DataItem>`.
+   * Returns the opaque `DataListResult` envelope. For runtime entities the
+   * handler builds a typed `Collection<DataItem, DataListXMedkit>` (SOVD
+   * ValueMetadata items with x-medkit ROS2 extensions plus typed fan-out via
+   * `fan_out_collection<DataItem>`) and serializes it into the envelope, so the
+   * wire shape is unchanged. For plugin entities the provider's free-form item
+   * shape (which may carry vendor per-item fields the gateway cannot describe at
+   * compile time) is passed through verbatim, instead of being re-parsed into
+   * `Collection<DataItem>` (which would silently drop those fields).
    */
-  http::Result<dto::Collection<dto::DataItem, dto::DataListXMedkit>> list_data(const http::TypedRequest & req);
+  http::Result<dto::DataListResult> list_data(const http::TypedRequest & req);
 
   /**
    * @brief Get a single data item (topic value) for an entity.

src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/registry.hpp

@@ -85,12 +85,18 @@ using AllDtos =
                VersionInfoEntry, XMedkitVersionInfo, VersionInfo, RootCapabilities, RootAuth, RootTls, RootOverview>;
 
 namespace detail {
+template <class T>
+void collect_one(nlohmann::json & schemas) {
+  static_assert(!dto_name<T>.empty(),
+                "collect_component_schemas: a DTO in AllDtos is missing a dto_name<T> specialization "
+                "(would write its schema under an empty \"\" key and collide with others)");
+  schemas[std::string(dto_name<T>)] = SchemaWriter<T>::schema();
+}
+
 template <class Tuple, std::size_t... I>
 nlohmann::json collect_impl(std::index_sequence<I...> /*seq*/) {
   nlohmann::json schemas = nlohmann::json::object();
-  ((schemas[std::string(dto_name<std::tuple_element_t<I, Tuple>>)] =
-        SchemaWriter<std::tuple_element_t<I, Tuple>>::schema()),
-   ...);
+  (collect_one<std::tuple_element_t<I, Tuple>>(schemas), ...);
   return schemas;
 }
 }  // namespace detail

src/ros2_medkit_gateway/include/ros2_medkit_gateway/dto/schema_writer.hpp

@@ -40,6 +40,9 @@ nlohmann::json variant_schema(std::index_sequence<I...> /*seq*/) {
 template <class U>
 nlohmann::json schema_of() {
   if constexpr (is_dto_v<U>) {
+    static_assert(!dto_name<U>.empty(),
+                  "schema_of: DTO type is missing a dto_name<T> specialization "
+                  "(would emit an empty \"$ref\" and a \"\" schema key)");
     return nlohmann::json{{"$ref", "#/components/schemas/" + std::string(dto_name<U>)}};
   } else if constexpr (is_optional_v<U>) {
     auto inner = schema_of<typename U::value_type>();
@@ -88,7 +91,16 @@ struct SchemaWriter {
           for (std::size_t i = 0; i < f.enum_count; ++i) {
             values.push_back(std::string(f.enum_values[i]));
           }
-          prop["enum"] = values;
+          // For optional members schema_of() yields {anyOf:[<inner>, {null}]};
+          // attach the enum to the non-null branch ([0]) so the nullable claim
+          // and the enum constraint agree (a top-level enum lacking "null" would
+          // reject the null that anyOf advertises). Required members get the
+          // enum at the top level.
+          if (prop.contains("anyOf") && prop["anyOf"].is_array() && !prop["anyOf"].empty()) {
+            prop["anyOf"][0]["enum"] = values;
+          } else {
+            prop["enum"] = values;
+          }
         }
         props[std::string(f.key)] = prop;
         if (f.presence == Presence::kRequired) {

src/ros2_medkit_gateway/include/ros2_medkit_gateway/http/handlers/handler_support.hpp

@@ -0,0 +1,64 @@
+// Copyright 2026 bburda
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+#pragma once
+
+#include <string>
+#include <type_traits>
+#include <utility>
+#include <variant>
+
+#include <nlohmann/json.hpp>
+
+#include "ros2_medkit_gateway/core/models/error_info.hpp"
+#include "ros2_medkit_gateway/http/handlers/handler_context.hpp"
+#include "ros2_medkit_gateway/http/typed_router.hpp"
+
+namespace ros2_medkit_gateway {
+namespace handlers {
+
+/// Build a SOVD-shaped ErrorInfo. Empty `params` are dropped so the wire body
+/// matches the legacy `send_error` default and integration tests stay byte-
+/// identical. Shared by every typed handler (was duplicated per handler TU).
+inline ErrorInfo make_error(int status, const std::string & code, std::string message, nlohmann::json params = {}) {
+  ErrorInfo err;
+  err.code = code;
+  err.message = std::move(message);
+  err.http_status = status;
+  if (!params.is_null() && !params.empty()) {
+    err.params = std::move(params);
+  }
+  return err;
+}
+
+/// Collapse a `validate_entity_for_route` validator error into a single
+/// `ErrorInfo`: the local-error branch passes through, while the `Forwarded`
+/// branch becomes the framework-internal sentinel that the RouteRegistry
+/// wrapper recognises and skips error rendering for. Shared by every typed
+/// handler (was duplicated per handler TU).
+inline ErrorInfo flatten_validator_error(const std::variant<ErrorInfo, http::Forwarded> & err) {
+  return std::visit(
+      [](auto && alt) -> ErrorInfo {
+        using T = std::decay_t<decltype(alt)>;
+        if constexpr (std::is_same_v<T, ErrorInfo>) {
+          return alt;
+        } else {
+          return HandlerContext::forwarded_sentinel_error();
+        }
+      },
+      err);
+}
+
+}  // namespace handlers
+}  // namespace ros2_medkit_gateway

src/ros2_medkit_gateway/src/http/handlers/bulkdata_handlers.cpp

@@ -35,6 +35,7 @@
 #include "ros2_medkit_gateway/dto/bulkdata.hpp"
 #include "ros2_medkit_gateway/dto/entities.hpp"
 #include "ros2_medkit_gateway/gateway_node.hpp"
+#include "ros2_medkit_gateway/http/handlers/handler_support.hpp"
 
 using json = nlohmann::json;
 
@@ -43,37 +44,6 @@ namespace handlers {
 
 namespace {
 
-/// Build a SOVD-shaped ErrorInfo. Empty `params` are dropped so the wire body
-/// matches the legacy `send_error` default and integration tests stay byte-
-/// identical.
-ErrorInfo make_error(int status, const std::string & code, std::string message, json params = {}) {
-  ErrorInfo err;
-  err.code = code;
-  err.message = std::move(message);
-  err.http_status = status;
-  if (!params.is_null() && !params.empty()) {
-    err.params = std::move(params);
-  }
-  return err;
-}
-
-/// Convert a ValidatorResult's error variant into a typed Result<T> error.
-/// When the validator returned Forwarded, the proxy already wrote the wire
-/// response, so the handler signals "do not render" via the framework-internal
-/// sentinel (ERR_X_INTERNAL_FORWARDED) the typed wrapper detects.
-ErrorInfo flatten_validator_error(const std::variant<ErrorInfo, http::Forwarded> & err) {
-  return std::visit(
-      [](auto && alt) -> ErrorInfo {
-        using T = std::decay_t<decltype(alt)>;
-        if constexpr (std::is_same_v<T, ErrorInfo>) {
-          return alt;
-        } else {
-          return HandlerContext::forwarded_sentinel_error();
-        }
-      },
-      err);
-}
-
 /// Resolve the entity_id from the typed request. Bulk-data routes embed the
 /// entity reference in the URL path; the registered route patterns capture
 /// the entity id as group 1 (single-entity) or group 2 (nested subarea /