feat(ssot)!: TOML render map, generated tdbe ticks, per-endpoint typed Greeks by userFRM · Pull Request #475 · userFRM/ThetaDataDx

userFRM · 2026-05-05T20:46:46Z

Summary

Closes the #474 SSOT gap end-to-end on a single PR.

TOML-driven render map. Every per-language binding name a renderer needs for one tick type lives in [types.X.render] blocks in crates/thetadatadx/tick_schema.toml. The 20 helper functions that previously hardcoded those names with parallel match dispatches collapsed into single HashMap lookups against a OnceLock-cached load of the schema. Adding a tick type is now a TOML edit -- no helper code touched.
Generated tdbe::types::tick structs. crates/tdbe/src/types/tick_generated.rs is emitted from the schema by generate_sdk_surfaces. Hand-written tick.rs keeps impl_contract_id! macro applications, TradeTick flag helpers, and OptionContract::is_call / is_put -- everything else flows from the schema. Per-field offset_of! asserts in tdbe::types::tick::layout_asserts pin every observable struct offset the FFI mirrors rely on.
Per-endpoint typed Greeks. The full union becomes GreeksAllTick (with new bid, ask, underlying_ms_of_day, underlying_price columns the upstream OpenAPI publishes); per-order endpoints return GreeksFirstOrderTick, GreeksSecondOrderTick, GreeksThirdOrderTick. C ABI gains four typed tdx_greeks_*_tick_array_free symbols.

Review-round 2: addressed 6 findings (1 HIGH compile failure in tools/mcp, 1 HIGH ABI guard gap, 4 lower).

Breaking

GreeksTick removed in every language and on the C ABI. Use GreeksAllTick for _greeks_all / _greeks_eod endpoints; per-order endpoints return their typed subset. Migration table in CHANGELOG.md under [8.0.26] -> Breaking.
tdx_greeks_tick_array_free -> tdx_greeks_all_tick_array_free (no forwarding shim).
GreeksAllTick field offsets shift by 16 bytes (bid + ask) vs the legacy GreeksTick. Rebuild any binary that links the C struct.

Validation

Local CI gate (all green on the final commit):

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace (462 tests pass, 0 failures, including the new 22 layout / per-field offset asserts in tdbe)
cargo deny check
cargo run -p thetadatadx --bin generate_sdk_surfaces --features config-file -- --check
cargo check --manifest-path tools/mcp/Cargo.toml
cargo clippy --manifest-path tools/mcp/Cargo.toml -- -D warnings

Test plan

CI green on cargo fmt --check.
CI green on cargo clippy --workspace -- -D warnings.
CI green on cargo test --workspace.
CI green on cargo deny check.
CI green on generate_sdk_surfaces --check.
FFI smoke (sdks/cpp + sdks/go) compiles against the renamed tdx_greeks_*_tick_array_free C symbols.
Python build picks up GreeksAllTick, GreeksFirstOrderTick, GreeksSecondOrderTick, GreeksThirdOrderTick pyclasses.
TypeScript build picks up the four new napi(object) interfaces in index.d.ts.

Closes #474

…efs #474) Every per-language binding name a renderer needs for one tick type -- Rust direct-client return type, generated parser fn, Go struct + converter, FFI array struct + free fn + output variant + header-return type, C++ value type, six Python converters (dict, columnar, pyclass-list, pyclass-list-class, vec-to-pylist, slice-arrow), TypeScript class + class-vec converter, plus the Python pyclass struct name -- now lives in `[types.X.render]` blocks in `crates/thetadatadx/tick_schema.toml`. The 20 helper functions that previously hardcoded those names with parallel `match` dispatches collapse into single HashMap lookups against a `OnceLock`-cached load of the schema: * `build_support/endpoints/helpers.rs::direct_return_type` / `direct_parser_name` / `go_result_type` / `go_converter_name` / `ffi_array_type` / `ffi_array_empty_expr` / `ffi_output_variant` / `ffi_from_vec_array_type` / `ffi_header_return_type` / `ffi_free_fn` / `cpp_value_type` / `cpp_converter_expr` / `python_converter` / `python_columnar_converter` / `python_pyclass_list_converter` / `python_pyclass_list_class` / `python_vec_to_pylist_converter` / `python_slice_arrow_converter` / `ts_class_name` / `ts_class_vec_converter` * `build_support/ticks/mod.rs::pyclass_name` Adding a tick type now requires one TOML row -- no helper edits. The generated SDK surfaces (sdks/{python,typescript,go,cpp}/...) are byte-identical against main because the new TOML rows reproduce the names the helpers previously hardcoded; the `--check` regen pass shows zero drift. Sets up the SSOT foundation for the per-endpoint typed Greeks structs and tdbe tick struct generation tracked in #474; those land in follow-up PRs to keep this change reviewable. CI gates locally: cargo fmt, cargo clippy --all-targets -D warnings, cargo test --workspace, cargo deny check, and `generate_sdk_surfaces --check` are all clean. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Drive the FFI sizes / layout asserts / Go converter dispatch / Python list-fn / TS class-vec / CLI raw-header constants from `tick_schema.toml` [types.X.render] entries instead of hand-coded match arms. Adding a tick type now requires zero edits in `build_support/ticks/{cpp,go,python_classes, typescript,cli_headers}.rs`. * `OpenInterestTick` and `TradeQuoteTick` gained `align = 64` in the schema -- they were `#[repr(C, align(64))]` in `tdbe::types::tick` but the schema row was missing the directive, so the Go/C++ FFI size emitters under-counted (32/144 vs the actual 64/192). The hand-coded emitters compensated by reading `size_of::<tdbe::T>()` directly; with the schema as the SSOT the directive must live there. * `cpp.rs` now reaches into `go::tick_ffi_size_and_align` for the schema-computed (size, align) pair instead of `size_of::<tdbe::T>()`, so the C++ static_asserts share one layout calculator with Go. * Dropped the dead `pub(crate) use ts_tick_class_factory_name` re-export (no remaining caller outside the typescript emitter). refs #474

`crates/tdbe/src/types/tick_generated.rs` is now emitted by `generate_sdk_surfaces` from the schema. Hand-written `tick.rs` `include!`s the generated file and keeps only the items the schema cannot express: `impl_contract_id!` macro applications, `TradeTick` flag helpers, and `OptionContract` is_call/is_put. Layout asserts in `types::tick::layout_asserts` pin every struct's size + alignment to the values the C / Go FFI mirrors and `tick_layout_asserts.hpp.inc` rely on, so a schema edit that drifts a layout fails `cargo test -p tdbe` before reaching the FFI side. refs #474

Copilot

Pull request overview

This PR advances the SSOT refactor for tick-type rendering by moving per-language binding symbol names into tick_schema.toml render blocks and replacing many per-tick match dispatch helpers with schema-driven lookups (cached via OnceLock). It also wires additional schema-driven generators (notably for tdbe tick structs) into the checked-in surface generation pipeline.

Changes:

Add [types.X.render] blocks to crates/thetadatadx/tick_schema.toml to centralize per-language binding names per tick type.
Replace hardcoded helper matches (Rust/Go/Python/TS/C++/FFI naming) with single-key render-map lookups.
Extend tick surface generation to emit crates/tdbe/src/types/tick_generated.rs and update related SDK/layout assertion generators.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
sdks/go/tick_ffi_sizes_generated.go	Regenerated Go expected-size constants ordering (schema-driven emission).
sdks/go/ffi_layout_generated_test.go	Regenerated Go FFI layout size test table ordering (schema-driven).
docs-site/docs/changelog.md	Changelog entry documenting TOML-driven render map refactor.
crates/thetadatadx/tick_schema.toml	Adds per-type `[render]` blocks containing binding names across languages; adds some `align` hints.
crates/thetadatadx/build_support/ticks/typescript.rs	Uses schema render map (`ts_class_vec`) instead of hardcoded matches for TS factory naming.
crates/thetadatadx/build_support/ticks/tdbe_structs.rs	New generator emitting `tdbe` tick struct definitions from the schema.
crates/thetadatadx/build_support/ticks/schema.rs	Adds `TickRenderDef` to schema model and `render_for_type` accessor.
crates/thetadatadx/build_support/ticks/python_classes.rs	Replaces Python naming match tables with schema render lookups.
crates/thetadatadx/build_support/ticks/mod.rs	Wires new generators (including `tdbe` tick structs) and updates `pyclass_name` to schema-driven mapping.
crates/thetadatadx/build_support/ticks/go.rs	Replaces Go converter dispatch matches with schema render map; adds FFI size+align helper.
crates/thetadatadx/build_support/ticks/cpp.rs	Drives C++ layout asserts from schema-derived FFI size/align.
crates/thetadatadx/build_support/ticks/cli_headers.rs	Derives CLI raw-header const identifiers programmatically instead of matching.
crates/thetadatadx/build_support/endpoints/render/mdds.rs	Minor template interpolation change to accommodate new String composition.
crates/thetadatadx/build_support/endpoints/helpers.rs	Introduces `OnceLock`-cached TOML render map and collapses many helpers to lookups.
crates/tdbe/src/types/tick.rs	Switches to `include!(\"tick_generated.rs\")` and adds layout size/align tests.
crates/tdbe/src/types/tick_generated.rs	New generated tick struct definitions file (checked in).
CHANGELOG.md	Changelog entry documenting TOML-driven render map refactor.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

userFRM · 2026-05-05T21:30:15Z

+    if type_name == "QuoteTick" {
+        out.push_str("    /// Pre-computed midpoint: `(bid + ask) / 2.0`.\n");
+        out.push_str("    pub midpoint: f64,\n");
+    }
+
+    if def.contract_id {
+        out.push_str("    /// Contract expiration (`YYYYMMDD`). Populated on wildcard queries, 0 otherwise.\n");
+        out.push_str("    pub expiration: i32,\n");
+        out.push_str("    /// Contract strike price (decoded to `f64`).\n");
+        out.push_str("    pub strike: f64,\n");
+        out.push_str("    /// Contract right (`'C'` = 67, `'P'` = 80 ASCII). 0 on single-contract queries.\n");
+        out.push_str("    pub right: i32,\n");
+    }


Fixed in 693582b: QuoteTick field-order drift.

userFRM · 2026-05-05T21:30:17Z

+ffi_array = "TdxOptionContractArray"
+ffi_array_empty = "TdxOptionContractArray { data: ptr::null_mut(), len: 0 }"
+ffi_output_variant = "OptionContracts"
+ffi_from_vec_array = "TdxOptionContractArray"
+ffi_header_return = "TdxOptionContractArray"
+ffi_free_fn = "tdx_option_contract_array_free"


Fixed in 693582b: ffi_array_empty null_mut.

userFRM · 2026-05-05T21:30:18Z

+        // OptionContractArray uses `*mut` pointers so the empty literal cannot
+        // share the generic ARRAY_EMPTY constant. Render the explicit struct
+        // literal instead, with the indentation the existing emitter expects.
+        return "TdxOptionContractArray {\n        data: ptr::null_mut(),\n        len: 0,\n    }"


Fixed in 693582b: helpers.rs OptionContracts null_mut special-case.

userFRM · 2026-05-05T21:30:27Z

+        GeneratedSourceFile {
+            // tdbe tick structs -- `#[repr(C, align(N))]` definitions
+            // emitted from the schema. Hand-written `tick.rs` `pub use`s
+            // them and adds the macro applications + custom impls the
+            // schema cannot express.
+            relative_path: "crates/tdbe/src/types/tick_generated.rs",
+            contents: tdbe_structs::render_tdbe_tick_structs(&schema),
+        },


Addressed by updating the PR description to reflect the merged scope: this PR now lands every #474 phase (TOML render map, generated tdbe ticks, per-endpoint typed Greeks). The follow-up-PR section is removed and the trailer reads Closes #474.

userFRM · 2026-05-05T21:30:19Z

+// ─────────────────────────────────────────────────────────────────────────────
+//  Layout asserts -- pin the generated struct sizes/alignments to the values
+//  the C / Go FFI mirrors and `tick_layout_asserts.hpp.inc` rely on. A schema
+//  edit that drifts a layout is caught here on `cargo test --workspace -p tdbe`
+//  before it lands on the FFI side.
+// ─────────────────────────────────────────────────────────────────────────────
+
+#[cfg(test)]
+mod layout_asserts {
+    use super::*;
+    use std::mem::{align_of, size_of};


Fixed in 693582b: offset_of asserts.

Split the union `GreeksTick` into the four shapes the v3 server actually emits across the `option_*_greeks_*` family. The full union becomes `GreeksAllTick` (returned by `_greeks_all` and `_greeks_eod`); the per-order endpoints (`_first_order`, `_second_order`, `_third_order`) return strict subsets matching the upstream OpenAPI -- no zero-default columns leak from one subset into another. `GreeksAllTick` adds `bid`, `ask`, `underlying_ms_of_day`, and `underlying_price` columns the OpenAPI publishes that the legacy `GreeksTick` was missing. `decode.rs::HEADER_ALIASES` gains `underlying_ms_of_day` -> `underlying_timestamp` so the wire Timestamp -> ms-of-day conversion flows through `row_number` on every Greeks endpoint. C ABI surface - Add `tdx_greeks_all_tick_array_free`, `tdx_greeks_first_order_tick_array_free`, `tdx_greeks_second_order_tick_array_free`, `tdx_greeks_third_order_tick_array_free`. Matching FFI array structs shipped on every binding. - Drop `tdx_greeks_tick_array_free`. Callers update to the typed free fn. No forwarding shim. Layout safety - Per-field `offset_of!` asserts in `crates/tdbe/src/types/tick.rs::layout_asserts` pin every observable Rust field offset that the C / Go FFI mirrors index into. Catches the field-order regression class that `size_of` checks miss (e.g. swapping two same-size fields keeps total size constant). - `QuoteTick.midpoint` now emits AFTER the `contract_id` triple in `tick_generated.rs` to match the legacy `tick.rs` order the C header and Go FFI mirror were already shipping. - Drop the unused `ffi_array_empty` row from the schema render block and `ffi_array_empty_expr` helper. The FFI emitter hard-codes `data: ptr::null()` for every `*const T`-pointer array; the row was dead and its `null_mut()` literal would have produced a const/mut mismatch if anyone wired it through. refs #474

Wire serializers for GreeksAllTick / GreeksFirstOrderTick / GreeksSecondOrderTick / GreeksThirdOrderTick onto the matching EndpointOutput variants emitted by the generator. The MCP binary referenced the deleted GreeksTick / EndpointOutput::GreeksTicks names and failed to build. Add explicit cargo check for tools/mcp and tools/server in the Extended Surfaces job so out-of-workspace binaries can no longer silently regress on a workspace rename.

Promote the hand-written tdbe layout asserts to a generator-emitted crates/tdbe/src/types/tick_layout_asserts_generated.rs included from tick.rs. Coverage now includes size_of / align_of AND offset_of! for every column on every tick struct in tick_schema.toml -- including EodTick, OhlcTick, TradeQuoteTick, IvTick, MarketValueTick, OpenInterestTick, PriceTick, CalendarDay, InterestRateTick, which previously only had size/align checks. Extend the C++ layout-assert emitter to mirror the same offset table into sdks/cpp/include/tick_layout_asserts.hpp.inc as static_assert(offsetof(...) == ...) lines so a Rust struct field reorder fails the C++ build at the same point it fails cargo test. Adding a tick type to tick_schema.toml now picks up Rust + Go + C++ field-offset coverage automatically; no helper edit required.

The legacy `("GreeksTick", "implied_volatility") => "IV"` special-case no longer fires after the type rename, so the four new per-order Greeks structs (`GreeksAllTick` / `GreeksFirstOrderTick` / `GreeksSecondOrderTick` / `GreeksThirdOrderTick`) fell through to the default `go_pascal_case` path and exposed `ImpliedVolatility` while the existing `IVTick` kept emitting `IV`. Replace the per-type arms with a single `(_, \"implied_volatility\") => \"IV\"` rule so every Greeks struct AND `IVTick` now expose the same Go field name. Renaming `IVTick.IV` was rejected as the larger source-breaking change; `IV` is the historical Go SDK convention since v3.

The new `parse_greeks_first_order_ticks`, `parse_greeks_second_order_ticks`, and `parse_greeks_third_order_ticks` parsers were only exercised by `is_empty()`-style coverage; the existing decode tests called `parse_greeks_all_ticks` against per-order column subsets, not the dedicated parsers. Add three tests that synthesise the exact wire shape each vendor endpoint publishes (column lists pinned to `items_option_snapshot_greeks_*_order` in `scripts/upstream_openapi.yaml`) and assert every decoded field -- including `bid`, `ask`, the underlying snapshot pair, and `date` -- against the input row's known values. A future regression in any field on any per-order parser surfaces here.

The captured `option_history_greeks_all` fixture exercises two `HEADER_ALIASES` rewrites -- `implied_vol` -> `implied_volatility` and `underlying_timestamp` -> `underlying_ms_of_day` -- but the test only asserted the decoded `delta` and `iv_error` values; either alias could silently regress to default-zero without any test failing. Anchor the decoded `implied_volatility`, `underlying_ms_of_day`, and `underlying_price` (plus `date`) against known first-row values from the captured payload so an alias-table breakage surfaces here, not at runtime in production traffic.

The new GreeksFirstOrderTick / GreeksSecondOrderTick / GreeksThirdOrderTick frames coverage in test_frames_arrow.rs and test_frames_polars.rs was compile-time only -- the `every_tick_type_has_*_impl` checks proved the trait was implemented but no test asserted column count, exact column order, dtype per column, or any decoded value. Add three Arrow tests + three Polars tests, one per per-order Greeks struct, that pin: * total column count, * exact column order, * dtype per column (Int32 / Float64 / Utf8 or polars `String`), * one row's value for every column read back through the typed Arrow / Polars getters. A schema reorder, dtype regression, or per-column emitter typo on build_support/ticks/rust_frames.rs now surfaces here at test time.

#474) Add a synthetic alias-decode test that drives `parse_greeks_all_ticks` with headers using ONLY the v3 server-side names (`implied_vol`, `underlying_timestamp`) and non-zero values (IV = 0.42, underlying ms-of-day = 34_200_000). A broken `implied_vol -> implied_volatility` alias would silently zero-default via `opt_float(None)` and the fixture-driven capture test cannot catch it because the recorded `first_row_implied_volatility` happens to be `0.0`.

CI gate `cargo test --manifest-path tools/mcp/Cargo.toml --locked` failed because `cargo check` (run locally) does not compile test code. Test target still imported the deleted `GreeksTick` and called the renamed `serialize_greeks_ticks`. Update sample helper to construct `GreeksAllTick` with the new `bid`/`ask`/`underlying_ms_of_day`/ `underlying_price` fields and rename two call sites.

userFRM and others added 3 commits May 5, 2026 22:46

userFRM requested a review from Copilot May 5, 2026 21:05

Copilot started reviewing on behalf of userFRM May 5, 2026 21:06 View session

Copilot AI reviewed May 5, 2026

View reviewed changes

userFRM changed the title ~~feat: TOML-driven render map for tick binding names (refs #474)~~ feat(ssot)!: TOML render map, generated tdbe ticks, per-endpoint typed Greeks May 5, 2026

userFRM added 8 commits May 5, 2026 23:45

userFRM merged commit 0992c92 into main May 5, 2026
31 checks passed

userFRM deleted the feat/474-ssot-typed-greeks branch May 5, 2026 23:10

This was referenced May 5, 2026

docs(changelog): add exchange rename bullet under [8.0.26] #478

Merged

refactor!: rename Contract.root/exp_date to symbol/expiration (closes #467) #484

Merged

refactor: rename root to symbol and exp_date to expiration to match v3 vendor surface #468

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(ssot)!: TOML render map, generated tdbe ticks, per-endpoint typed Greeks#475

feat(ssot)!: TOML render map, generated tdbe ticks, per-endpoint typed Greeks#475
userFRM merged 12 commits into
mainfrom
feat/474-ssot-typed-greeks

userFRM commented May 5, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

userFRM May 5, 2026

Uh oh!

userFRM May 5, 2026

Uh oh!

userFRM May 5, 2026

Uh oh!

userFRM May 5, 2026

Uh oh!

userFRM May 5, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

userFRM commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking

Validation

Test plan

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

userFRM May 5, 2026

Choose a reason for hiding this comment

Uh oh!

userFRM May 5, 2026

Choose a reason for hiding this comment

Uh oh!

userFRM May 5, 2026

Choose a reason for hiding this comment

Uh oh!

userFRM May 5, 2026

Choose a reason for hiding this comment

Uh oh!

userFRM May 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

userFRM commented May 5, 2026 •

edited

Loading