feat(sdk): add ScalarRecord/ObjectRecord for parser-controlled timestamps (#92)

* feat(sdk): add ScalarRecord/ObjectRecord for parser-controlled timestamps

Introduce ScalarRecord {optional<Timestamp> ts, vector<NamedFieldValue>}
and ObjectRecord {optional<Timestamp> ts, BuiltinObject} as the new
return types of the SchemaHandler parse_scalars / parse_object
callables.

When ts is nullopt the host uses the message receive time as before;
when set, the parser-provided timestamp is used instead — restoring
the ability to align rows with timestamps embedded in the payload
(e.g. ROS Header.stamp, JSON "timestamp" field, sensor capture time).

The default parse() now iterates the returned records and respects each
record's timestamp, also enabling multi-row output from a single payload
(useful for batch messages such as JSON arrays).

* refactor(sdk): drop multi-row vector on parse_scalars to keep one alloc per call

Review feedback: returning std::vector<ScalarRecord> meant every call
ended up doing two heap allocations in the common case (outer vector +
the fields vector inside the only record), even for single-row decoding
on a hot streaming path.

The parser-controlled timestamp was the actual ask from the client;
multi-row batch was added speculatively for JSON arrays and is not a
shipping requirement. Drop the outer vector and return a single
ScalarRecord; one allocation total, same shape as the pre-change API
plus the optional<Timestamp>.

Parsers that need to emit multiple rows per payload can stage a batch
API later if/when a concrete use case shows up — the SchemaHandler
shape leaves room for that without breaking ScalarRecord.

* chore: bump conan package version to 0.3.0

* chore: bump plugin ABI to v5

---------

Co-authored-by: Davide Faconti <dfaconti@aurynrobotics.com>

Author: pabloinigoblasco

CMakeLists.txt

@@ -183,7 +183,7 @@ endif()
 if(PJ_INSTALL_SDK)
   include(CMakePackageConfigHelpers)
 
-  set(PJ_PACKAGE_VERSION "0.2.1")
+  set(PJ_PACKAGE_VERSION "0.3.0")
   set(PJ_PACKAGE_CMAKE_DIR ${CMAKE_INSTALL_LIBDIR}/cmake/plotjuggler_core)
 
   install(EXPORT plotjuggler_coreTargets

cmake/PjAbiCheck.cmake

@@ -15,7 +15,7 @@
 #                          bumping PJ_ABI_VERSION for a major break).
 #
 # Baseline location: pj_base/abi/baseline.abi — the single source of
-# truth for what the v4 ABI looks like. The baseline is generated from
+# truth for what the current ABI looks like. The baseline is generated from
 # a canary plugin DSO (mock_data_source_plugin) whose symbol surface
 # exercises the full ABI header set via the SDK.
 #

cmake/PjPluginManifest.cmake

@@ -83,7 +83,7 @@ function(pj_emit_plugin_manifest TARGET)
 
   if(NOT ARG_ABI_MAJOR)
     # Matches PJ_ABI_VERSION in pj_base/plugin_data_api.h. Bump in lockstep.
-    set(ARG_ABI_MAJOR 4)
+    set(ARG_ABI_MAJOR 5)
   endif()
 
   # Track manifest edits so CMake reconfigures when the source changes.

conanfile.py

@@ -7,7 +7,7 @@
   plugin_sdk   — umbrella for plugin authors (base + dialog SDK + parser SDK)
   plugin_host  — umbrella for host loaders (data_source/parser/toolbox/dialog)
 
-A consuming Conan recipe declares e.g. `plotjuggler_core/0.1.0` and then:
+A consuming Conan recipe declares e.g. `plotjuggler_core/0.3.0` and then:
 
     find_package(plotjuggler_core REQUIRED COMPONENTS plugin_sdk)
     target_link_libraries(my_plugin PRIVATE plotjuggler_core::plugin_sdk)
@@ -27,7 +27,7 @@
 
 class PlotjugglerCoreConan(ConanFile):
     name = "plotjuggler_core"
-    version = "0.2.1"
+    version = "0.3.0"
     license = "MIT"
     url = "https://github.com/PlotJuggler/plotjuggler_core"
     description = "C++20 foundation libraries for PlotJuggler: storage engine, plugin SDK, plugin host loaders."

pj_base/include/pj_base/builtin_object_abi.h

@@ -9,12 +9,12 @@
  *
  * Canonical-object production (any concrete sdk::* type listed in
  * BuiltinObjectType — see pj_base/builtin/builtin_object.hpp) and the
- * pure-functional scalar production (Expected<vector<NamedFieldValue>>)
- * are C++ SDK contracts: plugins inheriting from MessageParserPluginBase
- * register handlers in SchemaHandler, and the in-process host consumes
- * them via MessageParserPluginBase::parseObject() and parseScalars()
- * called directly on the C++ pointer. Pure-C plugins emit scalars via
- * the parse() slot (writing to writeHost).
+ * pure-functional scalar production (Expected<ObjectRecord> /
+ * Expected<ScalarRecord>) are C++ SDK contracts: plugins inheriting from
+ * MessageParserPluginBase register handlers in SchemaHandler, and the
+ * in-process host consumes them via MessageParserPluginBase::parseObject()
+ * and parseScalars() called directly on the C++ pointer. Pure-C plugins
+ * emit scalars via the parse() slot (writing to writeHost).
  */
 // Copyright 2026 Davide Faconti
 // SPDX-License-Identifier: MIT

pj_base/include/pj_base/plugin_data_api.h

@@ -45,15 +45,20 @@ extern "C" {
  * (e.g. DataSource + Dialog in one .so) work without any extra ceremony.
  * Do not redefine it manually.
  *
- * v4 plugins advertise version 4. Data-plane changes from the pre-v4 design:
+ * v5 plugins advertise version 5. Relative to v4, this bump rejects
+ * pre-v5 parser DSOs because the C++ MessageParserPluginBase
+ * pure-functional contract changed parseScalars()/parseObject() return
+ * types to ScalarRecord/ObjectRecord.
+ *
+ * The C data-plane layout remains the v4 layout:
  *   - Arrow C Data Interface replaces Arrow IPC bytes at the boundary
  *     (append_arrow_stream + read_series_arrow).
  *   - append_arrow_ipc removed from all write hosts.
  *   - read_series (PJ_materialized_series_t) removed from toolbox host.
  *   - Every vtable slot is PJ_NOEXCEPT.
  *   - Every slot carries a thread-class tag (// [main-thread], etc.).
  */
-#define PJ_ABI_VERSION 4
+#define PJ_ABI_VERSION 5
 
 /**
  * Convention for plugin-loaders:
@@ -90,13 +95,13 @@ typedef enum {
   PJ_PRIMITIVE_TYPE_UNSPECIFIED = 0xFF,
 } PJ_primitive_type_t;
 
-/* ABI-FROZEN: layout permanent; changes = v4 break. */
+/* ABI-FROZEN: layout permanent; changes = ABI break. */
 typedef struct {
   const char* data;
   size_t size;
 } PJ_string_view_t;
 
-/* ABI-FROZEN: layout permanent; changes = v4 break. */
+/* ABI-FROZEN: layout permanent; changes = ABI break. */
 typedef struct {
   const uint8_t* data;
   size_t size;
@@ -162,17 +167,17 @@ struct ArrowArrayStream {
 
 #endif /* ARROW_C_DATA_INTERFACE */
 
-/* ABI-FROZEN: layout permanent; changes = v4 break. */
+/* ABI-FROZEN: layout permanent; changes = ABI break. */
 typedef struct {
   uint32_t id;
 } PJ_data_source_handle_t;
 
-/* ABI-FROZEN: layout permanent; changes = v4 break. */
+/* ABI-FROZEN: layout permanent; changes = ABI break. */
 typedef struct {
   uint32_t id;
 } PJ_topic_handle_t;
 
-/* ABI-FROZEN: layout permanent; changes = v4 break. */
+/* ABI-FROZEN: layout permanent; changes = ABI break. */
 typedef struct {
   PJ_topic_handle_t topic;
   uint32_t id;
@@ -506,7 +511,7 @@ typedef struct {
  * `pj.parser_object_write.v1` in addition to `pj.parser_write.v1`.
  * ========================================================================== */
 
-/* ABI-FROZEN: layout permanent; changes = v5 break. */
+/* ABI-FROZEN: layout permanent; changes = ABI break. */
 typedef struct {
   uint32_t id; /* 0 == invalid handle */
 } PJ_object_topic_handle_t;

pj_base/include/pj_base/sdk/plugin_data_api.hpp

@@ -14,6 +14,8 @@
 #include <variant>
 #include <vector>
 
+#include "pj_base/builtin/builtin_object.hpp"
+
 #include "pj_base/expected.hpp"
 #include "pj_base/plugin_data_api.h"
 #include "pj_base/sdk/arrow.hpp"
@@ -96,6 +98,25 @@ struct BoundFieldValue {
   ValueRef value;
 };
 
+/// One row of scalar data with an optional parser-controlled timestamp.
+/// When `ts` is nullopt the host uses the message's own timestamp
+/// (the value passed to parse_scalars). Set `ts` to override — e.g. to
+/// use a timestamp field embedded inside the payload rather than the
+/// transport-level receive time.
+struct ScalarRecord {
+  std::optional<Timestamp> ts;
+  std::vector<NamedFieldValue> fields;
+};
+
+/// A builtin object (image, point cloud, …) with an optional parser-controlled
+/// timestamp. Mirrors ScalarRecord for the object route: when `ts` is nullopt
+/// the host uses the message's own timestamp; set it to use the sensor time
+/// embedded in the payload (e.g. ROS Header.stamp).
+struct ObjectRecord {
+  std::optional<Timestamp> ts;
+  BuiltinObject object;
+};
+
 class CatalogSnapshot {
  public:
   CatalogSnapshot() = default;

pj_base/tests/abi_layout_sentinels_test.cpp

@@ -1,10 +1,10 @@
 /**
  * @file abi_layout_sentinels_test.cpp
- * @brief Compile-time sentinels that pin the v4 plugin ABI layout.
+ * @brief Compile-time sentinels that pin the plugin C ABI layout.
  *
  * Every assertion here is a static_assert. A failure at compile time means
  * a struct defined in the ABI-visible headers has shifted in a way that
- * would silently break binary compatibility with existing v4 plugins.
+ * would silently break binary compatibility with existing plugins.
  *
  * Maintenance rule:
  *   - Sizes and alignments are allowed to GROW at the tail (new slots
@@ -16,7 +16,8 @@
  *     break.
  *   - MIN-size constants (PJ_*_MIN_VTABLE_SIZE) MUST NEVER INCREASE
  *     within a major version. They are pinned at v4.0 release and are
- *     the floor that forward compatibility relies on within the v4 series.
+ *     the floor that forward compatibility relies on within the current
+ *     major series.
  *
  * Pinning target: x86-64 System V (Linux/macOS on Intel/AMD). For other
  * ABIs (ARM64, MSVC), either confirm identical layout during initial
@@ -63,7 +64,7 @@ static_assert(sizeof(PJ_borrowed_dialog_t) == 16, "PJ_borrowed_dialog_t fat poin
 
 // --- DataSource vtable (ABI-APPENDABLE within v4) ----------------------------
 // Offsets of v4.0 slots: PINNED. sizeof and MIN_VTABLE_SIZE are allowed to
-// grow at the tail via future appends within the v4 series.
+// grow at the tail via future appends within the current major series.
 static_assert(offsetof(PJ_data_source_vtable_t, protocol_version) == 0, "v4 prefix pinned");
 static_assert(offsetof(PJ_data_source_vtable_t, struct_size) == 4, "v4 prefix pinned");
 static_assert(offsetof(PJ_data_source_vtable_t, bind) == 40, "v4 bind slot pinned");
@@ -174,7 +175,7 @@ static_assert(offsetof(PJ_toolbox_host_vtable_t, read_series_arrow) == 64, "tool
 static_assert(sizeof(PJ_toolbox_host_vtable_t) == 72, "Toolbox host size");
 
 // --- ABI version symbol ------------------------------------------------------
-static_assert(PJ_ABI_VERSION == 4, "v4 ABI version");
+static_assert(PJ_ABI_VERSION == 5, "v5 ABI version");
 
 // This translation unit has no runtime behavior; the above are all
 // compile-time assertions. Linking only confirms the TU compiled.

pj_plugins/docs/ARCHITECTURE.md

@@ -1,9 +1,9 @@
 # Plugin System Architecture
 
-## 0a. ABI stability and evolution rules (v4)
+## 0a. ABI stability and evolution rules (v5)
 
 Seven rules the loader and every plugin author rely on. Breaking any of
-these is an ABI break and requires a v5 bump.
+these is an ABI break and requires a future `PJ_ABI_VERSION` bump.
 
 1. **Boot-level ABI symbol.** Every plugin .so exports
    `pj_plugin_abi_version` as a `uint32_t` symbol independent of any
@@ -18,13 +18,13 @@ these is an ABI break and requires a v5 bump.
    duplicate definitions across translation units in one DSO, so a single
    .so can host multiple plugin families (e.g. DataSource + Dialog) with
    one `PJ_*_PLUGIN(...)` macro per family — no duplicate-symbol error.
-   Current value is `PJ_ABI_VERSION == 4`.
+   Current value is `PJ_ABI_VERSION == 5`.
 
 2. **Min-vtable-size floor, pinned at v4.0.** Each family header defines
    `PJ_<FAMILY>_MIN_VTABLE_SIZE` — the byte count of the vtable as
    shipped in v4.0. The loader accepts
    `struct_size >= MIN_VTABLE_SIZE`. This constant MUST NEVER GROW
-   within the v4 series. Growing it would reject plugins compiled
+   within a major series. Growing it would reject plugins compiled
    against older v4 headers (which correctly report a smaller size),
    silently breaking the forward-compatibility promise.
 
@@ -40,7 +40,7 @@ these is an ABI break and requires a v5 bump.
    - **ABI-FROZEN**: `PJ_error_t`, `PJ_string_view_t`, `PJ_bytes_view_t`,
      `PJ_borrowed_dialog_t`, `PJ_service_t`, `PJ_service_registry_t`,
      handle types, primitive-value unions. Layout permanent; any change
-     is a v4 break. `PJ_error_t` has `extended` + `extended_kind` slots
+     is an ABI break. `PJ_error_t` has `extended` + `extended_kind` slots
      reserved as its one growth path — do not add further top-level
      fields.
    - **ABI-APPENDABLE**: all `*_vtable_t` types, service-host vtables,
@@ -101,10 +101,11 @@ for known ids or `nullptr`. Hosts call via `handle.getPluginExtension(id)`
 (tail-slot-gated). Use the experimental namespace for work-in-progress
 extensions; graduate to stable (`pj.<name>.v1`) once locked in.
 
-## 0. Protocol v4 (current)
+## 0. C protocol v4 (current under ABI v5)
 
-All four plugin families (DataSource, MessageParser, Toolbox, Dialog) track
-protocol v4. Key v4 distinguishing features (a superset of everything the
+All four plugin families (DataSource, MessageParser, Toolbox, Dialog) keep
+the v4 C protocol layouts under the v5 boot ABI. Key v4 distinguishing
+features (a superset of everything the
 previously-circulated pre-v4 design included):
 
 - **Arrow C Data Interface at the data boundary.** The write-host

pj_plugins/docs/data-source-guide.md

@@ -1,6 +1,6 @@
 # Writing a DataSource Plugin
 
-> **Tracks the v4 plugin ABI** (`PJ_ABI_VERSION == 4`). For the full
+> **Tracks the v5 plugin ABI** (`PJ_ABI_VERSION == 5`). For the full
 > evolution rules (tail-slot gating, MIN_VTABLE_SIZE, ABI-FROZEN vs
 > ABI-APPENDABLE structs, Arrow C Data Interface at the write boundary,
 > PJ_NOEXCEPT discipline) see `ARCHITECTURE.md`. This guide walks