Reference document for porting PlotJuggler plugins to the new SDK. Based on lessons learned porting the CSV plugin (the only fully verified plugin so far). Other plugins (Parquet, ULog, MCAP, Protobuf, ROS, ZMQ, MQTT, Foxglove Bridge, PJ Bridge) are ported but not yet validated end-to-end — expect similar issues.
Porting a plugin is a mechanical translation, not a rewrite. The original plugin is the specification. Your job is to produce code that behaves identically to the original in every observable way, using the new SDK as the target language. Nothing more, nothing less.
-
Read the original code in its entirety before writing anything. Not a skim — a line-by-line read. Open every function, every signal/slot connection, every conditional branch. You cannot translate what you have not read.
-
Every code path in the original must have a corresponding code path in the port. If the original has 14
ifbranches, the port has 14ifbranches. If the original emits a warning on a specific condition, the port emits the same warning on the same condition. If the original saves dialog geometry, the port saves dialog geometry. -
There are no minor features. Dialog geometry persistence is not minor. Splitter ratios are not minor. A help button that opens a sub-dialog is not minor. Column selection history is not minor. Every feature the user can observe, interact with, or benefit from is equally mandatory. Do not classify features by importance — classify them only as "ported" or "not yet ported."
-
Do not optimize, simplify, or "improve" the original's behavior. If the original has verbose logic, redundant checks, or a roundabout approach — translate it faithfully. The original was tested in production. Your "improvement" was not. If you believe something in the original is a bug, flag it explicitly — do not silently fix it.
-
Do not substitute a different approach. If the original uses
QTreeWidget, you do not get to decide thatQTableWidgetis "close enough." If the SDK does not support what the original uses, that is a blocker — see the escalation rules below. -
Speed does not matter. Completeness does. This work should take as many iterations as needed. Porting one plugin may take multiple sessions. That is expected and acceptable. What is not acceptable is declaring a plugin "ported" when features have been silently dropped.
If the new SDK lacks a capability that the original plugin uses (e.g., a widget type, an event type, a host callback), you have exactly two options:
- Option A: Extend the SDK to support it, then use it.
- Option B: Stop and ask the user what to do.
You do NOT have the option to silently drop the feature, substitute a simpler alternative, or mark it as a "minor gap." If you are unsure whether the SDK can do something, ask.
Before you say a plugin port is complete, you must produce a feature audit table with one row per feature of the original plugin:
| # | Original Feature | Original Code Location | Port Status | New Code Location | Notes |
|---|---|---|---|---|---|
| 1 | (feature name) | (file:line) | DONE / BLOCKED / NOT YET | (file:line) | (if BLOCKED: what SDK gap) |
Every row must be filled. Empty rows or missing features mean the audit is incomplete. The port is not done until every row says DONE or BLOCKED (with an explicit SDK limitation and a decision from the user on how to proceed).
These are patterns that have repeatedly caused incomplete ports. Treat them as red flags — if you catch yourself doing any of these, stop and correct:
- "This is just cosmetic" — Cosmetic features are features. Port them.
- "The SDK doesn't support X, so I'll use Y instead" — Stop. Ask.
- "I'll come back to this later" — You will not. Port it now or mark it BLOCKED with a specific SDK limitation.
- "The original code is messy, let me clean it up" — Do not. Translate it as-is. Cleanup is a separate task that can only happen after the translation is verified correct.
- "This edge case probably never happens" — The original author handled it. You handle it too.
- "I'm almost done, just these last few things" — Run the audit table. If any row is not DONE or BLOCKED, you are not almost done.
Header: pj_base/include/pj_base/sdk/data_source_patterns.hpp
Example: pj_plugins/examples/mock_file_source.cpp
Override:
uint64_t extraCapabilities() const— e.g.kCapabilityDirectIngest | kCapabilityHasDialogStatus importData()— do all work here;writeHost()andruntimeHost()are availablestd::string saveConfig() const/Status loadConfig(std::string_view json)
The base class manages the state machine automatically: Idle → Starting → importData() → Stopped/Failed.
Config convention: file importers receive {"filepath":"/path/to/file"} via loadConfig(). The manifest must include "file_extensions":[".csv",".tsv"] for the host file dialog.
Header: pj_base/include/pj_base/sdk/data_source_patterns.hpp
Example: pj_plugins/examples/mock_source_with_dialog.cpp
Override:
uint64_t extraCapabilities() constStatus onStart()— open connectionsStatus onPoll()— must not block; drain available data and returnvoid onStop()— must be idempotent; close connections
Header: pj_base/include/pj_base/sdk/message_parser_plugin_base.hpp
Example: pj_plugins/examples/mock_json_parser.cpp
Override:
Status parse(Timestamp ts, Span<const uint8_t> payload)— pure virtualStatus bindSchema(std::string_view type_name, Span<const uint8_t> schema)— optional
Manifest must include "encoding" key (e.g. "json", "cdr", "protobuf").
kCapabilityFiniteImport // auto-added by FileSourceBase
kCapabilityContinuousStream // auto-added by StreamSourceBase
kCapabilityDirectIngest // plugin writes data via writeHost()
kCapabilityDelegatedIngest // plugin pushes raw bytes to host-side parsers
kCapabilityHasDialog // plugin provides a dialog UI
kCapabilitySupportsPause // must override pause()/resume()PJ_DATA_SOURCE_PLUGIN(ClassName, R"({"name":"...","version":"1.0.0"})")
PJ_MESSAGE_PARSER_PLUGIN(ClassName, R"({"name":"...","version":"1.0.0","encoding":"..."})")
PJ_DIALOG_PLUGIN(DialogClassName, kMyManifest) // if plugin has a dialog (pass the embedded manifest)Header: pj_base/include/pj_base/sdk/plugin_data_api.hpp
using ValueRef = std::variant<NullValue,
float, double,
int8_t, int16_t, int32_t, int64_t,
uint8_t, uint16_t, uint32_t, uint64_t,
bool, std::string_view>;PITFALL (hit during porting): The initial protobuf and JSON parsers cast everything to double, losing precision for int64/uint64 values. Push native types directly:
// WRONG — loses precision for large integers
out.push_back({name, static_cast<double>(value.get<int64_t>())});
// CORRECT — preserves full precision
out.push_back({name, value.get<int64_t>()});struct NamedFieldValue {
std::string name; // owned string (was string_view, changed for safety)
ValueRef value;
};The name field is now std::string (previously string_view). This eliminates the dangling reference foot-gun that affected early plugin ports. You can safely build field names via string concatenation:
std::vector<PJ::sdk::NamedFieldValue> fields;
fields.push_back({prefix + "/" + key, value}); // safe — name is owned
writeHost().appendRecord(ts, PJ::Span(fields.data(), fields.size()));Store raw timestamps (e.g. epoch nanoseconds) in the datastore. Never subtract a base time during ingestion. Display-time subtraction belongs in the UI layer only.
// Epoch seconds from CSV → nanoseconds for datastore
auto ts = static_cast<int64_t>(epoch_seconds * 1e9);
writeHost().appendRecord(*topic, PJ::Timestamp{ts}, fields);File: pj_datastore/src/writer.cpp, lines 516-567
ensureColumn() auto-seals the current chunk when a new column is added
after rows already exist. The expanded column set continues in a fresh chunk.
Readers treat absent columns in earlier chunks as null.
Pre-registration is optional but recommended when the schema is known
upfront — it avoids mid-stream chunk sealing and enables the faster
bound-write path (appendBoundRecord).
auto topic = writeHost().ensureTopic(topic_name);
// Optional: pre-register fields when the schema is known upfront
for (const auto& col_name : all_column_names) {
writeHost().ensureField(*topic, col_name, PJ::PrimitiveType::kFloat64);
}
// Write data — fields auto-create on first non-null value if not pre-registered
for (...) {
writeHost().appendRecord(*topic, ts, fields);
}The original CSV segfault was caused by the reader not handling varying column sets across chunks. That is now fixed — chunks may have different column sets.
File: pj_datastore/src/chunk.cpp, lines 165-180
When appendRecord() is called with only some fields set, finishRow() automatically pads unset columns with null (zero bytes + validity bit cleared). This is correct behavior — sparse records are supported.
Example: topic has columns [x, y, z]. Record sets only x:
- Column x: value written
- Column y: null (zero bytes, validity bit = 0)
- Column z: null (zero bytes, validity bit = 0)
All columns always have the same row count. This is an invariant.
File: pj_datastore/src/chunk.cpp, lines 664-705
readColumnAsDoubles() reads raw bytes and converts to double. For null rows, it now returns NaN (not 0.0). This means consumers don't need to manually check isNull() — NaN values naturally propagate and are not plotted by chart panels.
range.chunk->readColumnAsDoubles(col_index, values_span, row_start);
for (size_t i = 0; i < row_count; ++i) {
if (std::isnan(values[i])) {
continue; // null value — skip
}
// use values[i]
}Note: readNumericAsDouble() (single-value read) does NOT check nulls — use isNull() if needed.
If columns were added after some chunks were sealed, early chunks have fewer columns. Before calling readColumnAsDoubles(col_index, ...), check bounds:
if (col_index >= range.chunk->columns.size()) {
return; // column doesn't exist in this chunk
}Pattern: pj_plugins/examples/mock_source_with_dialog.cpp
WidgetData API: pj_plugins/dialog_protocol/include/pj_plugins/sdk/widget_data.hpp
The dialog is part of the plugin's user-facing behavior. It must be ported with the same exactness as the data logic. Specifically:
- Copy the original
.uifile exactly. Only modify what strictly cannot work (e.g., custom widget classes whose headers don't exist in this codebase). Everything else — layout, sizing, spacing, naming, properties, tab order — stays identical. - Use real
.uifiles editable in Qt Creator, auto-embedded into aconstexpr char[]header via CMake'spj_embed_ui(). Do NOT write inline XML strings manually. Plugins must NOT depend on Qt for UI loading. - Smart dialog plugins: streaming source dialogs that have dynamic topic discovery (PJ Bridge, Foxglove, MQTT) must own the connection logic inside the dialog plugin itself, replicating the original Connect → discover → select → subscribe flow.
- Every signal/slot connection in the original dialog must be replicated.
If the original wires
itemDoubleClicked→accept(), the port must do the same. If the SDK lacks the event type, that is a blocker — extend the SDK or ask the user.
| Widget | set* Methods |
|---|---|
| QLineEdit | setText, setPlaceholder, setReadOnly |
| QComboBox | setItems, setCurrentIndex |
| QCheckBox / QRadioButton | setChecked |
| QSpinBox | setValue(int), setRange(min, max) |
| QDoubleSpinBox | setValue(double) |
| QListWidget | setListItems, setSelectedItems (vector of strings) |
| QTableWidget | setTableHeaders, setTableRows (vector of vector of strings) |
| QLabel | setLabel (also responds to setText) |
| QPushButton | setButtonText |
| QDialogButtonBox | setOkEnabled |
| QTabWidget | setTabIndex |
| Any widget | setEnabled, setVisible |
If the original plugin uses a widget type not in the table above (e.g., QTreeWidget, QTextEdit, QTableView, custom widgets), you must not silently replace it with a supported widget. Instead:
- Check if the SDK can be extended to support it (see "Extending the Dialog Protocol" below).
- If yes, extend it.
- If you are unsure, ask the user.
pj_plugins/dialog_protocol/ may need to be extended to support interactions that ported plugins require but the current SDK doesn't cover. Before adding new features, verify that the existing interface truly cannot accomplish the goal — sometimes a workaround using supported widgets is sufficient.
Known gaps where protocol extension may be needed:
- Double-click on list item to accept — old CSV plugin wires
listWidgetSeries::itemDoubleClicked→accept(). The dialog SDK has noonItemDoubleClickedevent. Could be added as a new event type inwidget_event.hpp. - QTextEdit / rich text — no binding for QTextEdit exists in
widget_binding.cpp. Could add one following the QLineEdit pattern. - Raw text preview with syntax highlighting — would require either QTextEdit support or a custom widget type.
- User-choice message boxes (Continue/Abort) —
reportMessageis fire-and-forget with no return value. A blocking prompt mechanism would require a new runtime host callback (e.g.confirmWarning(message) → bool).
When extending, modify these files:
pj_plugins/dialog_protocol/include/pj_plugins/sdk/widget_event.hpp— new event typespj_plugins/dialog_protocol/src/widget_binding.cpp— add widget type handlingpj_plugins/dialog_protocol/include/pj_plugins/sdk/widget_data.hpp— new set* methodspj_base/include/pj_base/data_source_protocol.h— for new runtime host callbacks
- Subclass
PJ::DialogPluginTyped - Use a real
.uifile (editable in Qt Creator) in the plugin'sui/directory. CMake auto-generates a header viapj_embed_ui()—ui_content()returns the generatedconstexpr char[]. No Qt dependency needed. Do NOT use inline XML strings.include(${CMAKE_CURRENT_LIST_DIR}/../cmake/EmbedUi.cmake) pj_embed_ui(my_plugin UI_FILE ${CMAKE_CURRENT_SOURCE_DIR}/ui/dialog.ui HEADER ${CMAKE_CURRENT_BINARY_DIR}/generated/dialog_ui.hpp VAR_NAME kDialogUi )
- QDialogButtonBox must have
name="buttonBox"andstandardButtonsproperty set:The DialogEngine searches for<widget class="QDialogButtonBox" name="buttonBox"> <property name="standardButtons"> <set>QDialogButtonBox::Cancel|QDialogButtonBox::Ok</set> </property> </widget>
findChild<QDialogButtonBox*>("buttonBox")to wire accept/reject signals. - DataSource owns dialog as member:
CsvDialog dialog_; - Override
dialogContext():return &dialog_; - Export with
PJ_DIALOG_PLUGIN(CsvDialog, kCsvManifest)alongsidePJ_DATA_SOURCE_PLUGIN
File: pj_proto_app/src/main_window.cpp, lines 155-175
- Host checks
capabilities & kCapabilityHasDialog - Loads config with filepath BEFORE showing dialog — so the dialog can analyze the file:
temp_handle.loadConfig(R"({"filepath":"/path/to/file"})"); - Gets
dialogContext()→ createsDialogEngine→ callsshowDialog(parent) - On accept: retrieves
savedConfig()and passes toloadConfig()for import
PITFALL (hit during porting): If you don't load the filepath config before showing the dialog, the dialog appears empty (no columns, no preview) because it doesn't know which file to analyze.
File: pj_proto_app/src/main_window.cpp, onLoadFile()
The host persists the plugin's last-used config in QSettings so dialogs remember user choices across sessions:
-
Before dialog: Restore saved config, merge in new filepath:
auto saved = settings.value("PluginConfig/<plugin_name>").toString().toStdString(); auto cfg = nlohmann::json::parse(saved); // last session's choices cfg["filepath"] = new_filepath; // override with current file temp_handle.loadConfig(cfg.dump()); // pre-populate dialog
-
After successful import: Save the config:
settings.setValue("PluginConfig/<plugin_name>", QString::fromStdString(config));
This preserves: delimiter choice, time column selection, custom format, etc.
QDialogButtonBoxmust be named"buttonBox"(not"button_box") — the DialogEngine searches by this exact nameQDialogButtonBoxmust havestandardButtonsproperty set in the XML, or no OK/Cancel buttons appearQTextEditis not supported by the widget binding system — see "When the original uses an unsupported widget" abovesetSelectedItemstakes a vector of strings (not indices) — use the item text, not a numeric index
- One topic per file (named after file basename without extension)
- Each CSV column becomes a named field in that topic
- The time column is excluded from data fields (used only for timestamps)
The turtle.csv dataset has sparse data: turtle1 columns are empty on rows where turtle2 has data. The correct approach:
- Pre-register columns (optional but recommended since CSV headers are known upfront)
- Group data points by timestamp (merge across columns)
- For each timestamp, write a record with only the columns that have data
- Missing columns are auto-filled with null
When user selects a time column via dialog:
- CSV parser uses that column's values as timestamps (detected type: EPOCH_SECONDS, DATETIME, etc.)
- Timestamps are converted to absolute nanoseconds:
int64_t(seconds * 1e9) - Time column itself is excluded from data fields
When no time column selected (row number mode):
- Row index is used as timestamp (0, 1, 2, ... in nanoseconds)
./build/pj_proto_app/pj_proto_app \
--plugin-dir ./build/pj_ported_plugins/bin/ \
--load /path/to/file.csv \ # auto-load file (skips dialog)
--plot 3 \ # auto-plot first N fields
--screenshot /tmp/output.png # take screenshot and exit./build.sh --debug # builds with ASAN in build/debug_asan/
./build/debug_asan/pj_proto_app/pj_proto_app \
--plugin-dir ./build/debug_asan/pj_ported_plugins/bin/ \
--load turtle.csv --plot 3 --screenshot /tmp/test.png
# Check stderr for ASAN errorsAll plugin .so files are built into a single directory for easy loading:
build/pj_ported_plugins/bin/ # set by CMAKE_LIBRARY_OUTPUT_DIRECTORY in umbrella CMakeLists.txt
This is set in pj_ported_plugins/CMakeLists.txt:
set(CMAKE_LIBRARY_OUTPUT_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/bin)~/ws_plotjuggler/src/PlotJuggler/datasamples/turtle.csv— sparse CSV with epoch timestamps, 563 rows, 13 columns. Time range ~20.6 seconds. Good test for sparse data handling./tmp/test_simple.csv— create with:echo -e "time,x,y\n0,1.0,2.0\n1,3.0,4.0\n2,5.0,6.0"for quick smoke tests
./build.sh # RelWithDebInfo
./build.sh --debug # Debug + ASAN
./test.sh # all unit tests (both build dirs)
# Smoke test: load CSV, auto-plot, screenshot
./build/pj_proto_app/pj_proto_app \
--plugin-dir ./build/pj_ported_plugins/bin/ \
--load ~/ws_plotjuggler/src/PlotJuggler/datasamples/turtle.csv \
--plot 3 --screenshot /tmp/test.png
# ASAN smoke test (catches heap overflows, use-after-free)
./build/debug_asan/pj_proto_app/pj_proto_app \
--plugin-dir ./build/debug_asan/pj_ported_plugins/bin/ \
--load ~/ws_plotjuggler/src/PlotJuggler/datasamples/turtle.csv \
--plot 3 --screenshot /tmp/test_asan.png
# If ASAN reports errors, check stderrThe CSV plugin was audited feature-by-feature against the old plugin. 45 features ported, 6 partial, 12 missing.
| Feature | Status | Description |
|---|---|---|
| String series | FIXED | String-only columns now written via PrimitiveType::kString. Both numeric and string columns are pre-registered and included in the merged timeline. |
| Non-monotonic time warning | FIXED | Warning reported via reportMessage(kWarning, ...). Note: old plugin showed Continue/Abort QMessageBox — a true blocking prompt requires extending the runtime host protocol (see Dialog Protocol section). |
| Skipped lines detail | FIXED | Per-line detail now reported via reportMessage(kWarning, "Line N: reason\n..."). Old plugin showed a QMessageBox with detailed text. |
| Double-click column to accept | SDK LIMITATION | Dialog SDK has no onItemDoubleClicked event. Requires protocol extension. |
| Column history | NOT PORTED | Old persists last-50 time column selections via QSettings. Would need dialog-side QSettings or host-provided persistence. |
| Date Format Help dialog | NOT PORTED | Old has a separate dialog with format reference tables. Could be implemented as a second dialog or inline help text. |
| Raw text preview tab | NOT PORTED | Old uses QCodeEditor with CSV syntax highlighting. Dialog SDK doesn't support QTextEdit. Requires protocol extension. |
| Feature | Description | What must happen |
|---|---|---|
| toDouble locale handling | Old uses QLocale::c() after replacing all commas. New uses strtod with different single-comma logic. | Must match original behavior exactly, including for inputs like "1,000,000". |
| Dialog geometry | Old saves/restores dialog geometry via QSettings. New relies on dialog SDK defaults. | Must be ported — either extend dialog SDK or replicate via config. |
| Splitter ratio | Old sets explicit stretch factors (1:2). New uses default. | Must be ported. |
Windows path separator: FIXED — now usesfind_last_of("/\\")to handle both separators.- Time column skipped by name not index: If two columns have the same raw name (before dedup), the wrong one could be skipped. Unlikely but fragile — dedup ensures unique names.
Preview label says "20 rows": FIXED — now says "100 rows" matching the actual code.
The following plugins compile and have unit tests but have NOT been tested end-to-end in pj_proto_app. They likely have similar issues to those found in the CSV plugin:
| Plugin | Risk | Likely Issue |
|---|---|---|
| DataLoadParquet | Medium | May need pre-registration of columns; timestamp precision loss (double→int64) |
| DataLoadULog | Medium | Same pre-registration issue possible; binary format edge cases |
| DataLoadMCAP | Low | Uses delegated ingest (no direct column writes) |
| ParserProtobuf | Low | Unit tested; native types now preserved |
| ParserROS | Medium | No unit tests; rosx_introspection API may have changed |
| DataStreamZMQ | Medium | strtod on non-null-terminated buffer was fixed; only parses text doubles |
| DataStreamMQTT | Medium | Encoding hardcoded (now configurable); binding cache added |
| DataStreamFoxgloveBridge | High | parser_encoding was wrong (fixed: ch.encoding not ch.schema_encoding) |
| DataStreamPJBridge | Medium | ZSTD decompression untested with real data |
- Pre-register columns when the schema is known upfront (recommended for direct ingest, optional otherwise)
- ValueRef types — don't cast to double; push native int64/uint64/bool
- string_view lifetime — ensure owned strings outlive appendRecord call
- Null handling — readers must check
isNull()before using values - Non-null-terminated buffers — copy to
std::stringbeforestrtod/parsing - Delegated ingest encoding —
parser_encodingmust be the wire format (e.g."cdr"), not the schema format (e.g."ros2msg")