Skip to content

feat(v9.1.0)!: streaming SSOT + fluent contract-first API + typed control variants + FLATFILES SDK parity#515

Merged
userFRM merged 1 commit into
mainfrom
fix/h1-streaming-ssot
May 9, 2026
Merged

feat(v9.1.0)!: streaming SSOT + fluent contract-first API + typed control variants + FLATFILES SDK parity#515
userFRM merged 1 commit into
mainfrom
fix/h1-streaming-ssot

Conversation

@userFRM
Copy link
Copy Markdown
Owner

@userFRM userFRM commented May 9, 2026

Summary

v9.1.0 single-queue SSOT for the FPSS streaming pipeline, polymorphic contract-first subscription API, typed FpssControl::* variants on every binding, hand-written FLATFILES surface across Python / TypeScript / C++, dynamic-schema Arrow conversion, and resilience hardening across the FPSS lifecycle. The Go SDK is dropped; Python, TypeScript, C, and C++ are first-class.

Streaming pipeline

  • One queue. The user callback runs as the LMAX Disruptor consumer's handle_events_with closure under std::panic::catch_unwind. The legacy dispatcher shim and crossbeam-channel runtime dep are gone.
  • Pull-iter deliverystart_streaming_iter() (Rust / Python / C++), streaming_iter() Python context manager, for await async iterator (TypeScript), TdxFpssEventIterator (C / C++).
  • Drain barrierawait_drain(timeout) / tdx_*_await_drain quiescence; tdx_*_free polls it (5 s) before destroying the handle.
  • Stop / restart — generation token closes stop_streaming() resurrection race; multi-generation drain via prev_drained: Mutex<Vec<Arc<AtomicBool>>>.
  • Reconnecttdx_unified_reconnect and tdx_fpss_reconnect block on the previous-generation drain flag (5 s budget) before binding the new session to the same C callback / ctx.
  • Self-join detachFpssClient::Drop detects the consumer-thread self-join case via OnceLock<ThreadId> and detaches io_handle.join() onto a helper thread.
  • Config knobsFpssConfig exposes ring_size, flush_mode, tcp_nodelay, tcp_keepalive, connect_timeout, read_timeout on the client builder.

API fluency

  • One polymorphic subscribe(spec) / unsubscribe(spec) method on the unified ThetaDataDxClient. Build a typed spec via quote() / trade() / open_interest() on Contract, or full_trades() / full_open_interest() on SecType. subscribe_many / unsubscribe_many for bulk.
  • Contract::stock("AAPL"), Contract::index("SPX"), Contract::option(symbol, expiration, strike, right). All strings positional in C++ / Rust; keyword-only in Python.
  • Event-carried Contract is fluent — event.contract exposes right, strike_dollars, sec_type (symbolic name), and per-binding-idiomatic shapes for the option side. C++ inline helpers tdx::strike_dollars, tdx::right, tdx::sec_type_name, tdx::reason_name mirror the Python / TS fluent fields.

Typed control variants

Every FpssControl::* Rust variant has a dedicated typed surface:

  • Python: one pyclass per variant (LoginSuccess, ContractAssigned, Disconnected, Reconnecting, ...). Branch on event.kind (snake_case), read the variant's typed payload directly. Disconnected.reason_name / Reconnecting.reason_name surface the RemoveReason enum name.
  • TypeScript: one #[napi(object)] struct per variant; event.disconnected.reasonName mirrors the Python surface.
  • C / C++: one typedef struct { ... } TdxFpss<Variant>; per variant. Dispatch on event->kind (TdxFpssEventKind enum), read the matching event-><variant> payload.

FpssControl::ContractAssigned keeps its diagnostic role; the resolved typed Arc<Contract> is carried directly on every data event, so user code never has to thread a contract-id side table.

FLATFILES SDK parity

Hand-written FLATFILES bindings on Python, TypeScript, and C++ — every endpoint the Rust crate exposes is reachable. Dynamic per-(sec_type, req_type) schema converted to Arrow under the arrow feature; Python returns pyarrow.Table, TypeScript returns Arrow IPC bytes, C++ returns TdxFlatFileRowList with to_arrow_ipc(). MCP server ships tdx_flatfile_* tool surfaces matching the SDK shape.

Cross-language utility parity

condition_name, exchange_name, sequence_signed_to_unsigned, sequence_unsigned_to_signed exposed across Python, TypeScript, and C++ — same names, same return types. TypeScript helpers reject out-of-range BigInt inputs at the i32 / u32 wire-range boundary instead of silently coercing.

Encoding crate (tdbe)

  • mdds::decode::v3 renamed to mdds::decode::dual_type_columns. Schema bumped to v5.
  • Error enum folded Decode / Decompress / Config / Grpc payloads into typed kinds; ~120 callsites migrated.
  • RemoveReason::from_code(i16) -> Self and RemoveReason::as_str(&self) -> &'static str for the symbolic-name accessors used by Python / TS bindings.
  • Canonical Gregorian calendar validator across MDDS, FPSS, and flat-files.

Repository / CI

  • Root tree trimmed; ADRs inlined into Rust doc comments; generated SDK files moved under _generated/ subdirectories.
  • TypeScript npm test runs all 6 test files (19 tests pass) on every advertised platform.
  • Stale RawData / Empty event variants hidden from the public surface; contract_id integer removed from all data events.
  • Observability path surfaces SystemTime and JSON-serialization failures instead of swallowing them.

Breaking changes since v9.0.x

Surface Before After
Client type name ThetaDataDx ThetaDataDxClient
Subscribe API client.subscribe_quotes(symbol) / subscribe_full_trades(sec_type) and the per-kind family client.subscribe(Contract::stock("AAPL").quote()) / client.subscribe(SecType::Option.full_trades())
Option builder Contract::option(symbol, occ_string) (2-arg) Contract::option(symbol, expiration, strike, right) (4-arg, all strings)
C ABI Contract option side bool has_is_call; bool is_call; bool has_right; char right; (ASCII 'C' / 'P')
Python / TS event-Contract sec_type: int, is_call: Optional[bool] sec_type: str, right: Optional[str], plus strike_dollars
Control envelope flat TdxFpssControl { kind, id, detail } per-variant typed structs (TdxFpssLoginSuccess, TdxFpssDisconnected, ...)
Data event contract id contract_id: i32 plus side-table lookup typed Arc<Contract> directly on the event
Hidden internals FpssData::RawData / FpssData::Empty public hidden from user surface
Go SDK shipped removed end-to-end

cargo semver-checks --baseline-rev v9.0.0 reports the breaking set; the release lands as a 9.x minor on the v9 line.

Closes

Test plan

  • cargo fmt --all -- --check
  • cargo clippy --workspace --locked -- -D warnings
  • cargo test --workspace --locked
  • cargo deny check all
  • cargo build --release -p thetadatadx-ffi --locked
  • C++ CMake build (cpp library, examples, validate, fpss_smoke, flatfiles)
  • TypeScript npm run build + npm test (19/19 pass)
  • Python cargo check + ABI3 smoke on macOS / Linux / Windows × py3.9, py3.14
  • python3 scripts/check_docs_consistency.py exits 0
  • cargo run -p thetadatadx --bin generate_sdk_surfaces -- --check reports no drift
  • cargo semver-checks --baseline-rev v9.0.0 — fails with the documented intentional-break set, not blocking
  • Live FPSS smoke against production (run by maintainer with creds before tag)

…trol variants + FLATFILES SDK parity

Single-queue SSOT for the FPSS streaming pipeline, polymorphic
contract-first subscription API, typed `FpssControl::*` variants on
every binding, hand-written FLATFILES surface across Python /
TypeScript / C++, dynamic-schema Arrow conversion, and resilience
hardening across the FPSS lifecycle. The Go SDK is dropped; Python,
TypeScript, C, and C++ are first-class.

This is a v9.1.0 minor release with breaking API changes; the v9.0
line had no external consumers, so the release rolls forward as a
9.x minor instead of bumping to 10.

# Streaming pipeline

- One queue. The user callback runs as the LMAX Disruptor consumer's
  `handle_events_with` closure under `std::panic::catch_unwind`. The
  legacy dispatcher shim and `crossbeam-channel` runtime dep are gone.
- **Pull-iter delivery** — `start_streaming_iter()` (Rust / Python /
  C++), `streaming_iter()` Python context manager, `for await` async
  iterator (TypeScript), `TdxFpssEventIterator` (C / C++). Mutually
  exclusive with the push-callback path on the same client; the
  iterator drains the SPSC queue from the user thread.
- **Drain barrier** — `await_drain(timeout)` /
  `tdx_*_await_drain` quiescence; `tdx_*_free` polls it (5 s) before
  destroying the handle, closing the `ctx` use-after-free window.
- **Stop / restart** — `stop_streaming()` resurrection race closed
  with a generation token; multi-generation drain via
  `prev_drained: Mutex<Vec<Arc<AtomicBool>>>` walks every prior
  generation's flag.
- **Reconnect** — `tdx_unified_reconnect` and `tdx_fpss_reconnect`
  block on the previous-generation drain flag (5 s budget) before
  binding the new session to the same C callback / `ctx`. The
  documented "single-thread callback" contract is now actually
  enforced by the FFI.
- **Self-join detach** — `FpssClient::Drop` detects the
  consumer-thread self-join case via `OnceLock<ThreadId>` and
  detaches `io_handle.join()` onto a helper thread.
- **Config knobs** — `FpssConfig` exposes `ring_size`, `flush_mode`,
  `tcp_nodelay`, `tcp_keepalive`, `connect_timeout`, `read_timeout`
  on the client builder. TypeScript shape-manifest agreement gap
  closed.

# API fluency on every binding

- **Subscriptions** — one polymorphic `subscribe(spec)` /
  `unsubscribe(spec)` method on the unified `ThetaDataDxClient`. Build
  a typed spec via `quote()` / `trade()` / `open_interest()` on
  `Contract`, or `full_trades()` / `full_open_interest()` on
  `SecType`. `subscribe_many` / `unsubscribe_many` for bulk. The
  legacy per-kind family is gone.
- **Contract builder** — `Contract::stock("AAPL")`,
  `Contract::index("SPX")`, `Contract::option(symbol, expiration,
  strike, right)` (all strings positional in C++ / Rust; keyword-only
  in Python). All four arguments required.
- **Event-carried Contract is fluent** across languages — `event.contract`
  exposes:
  - **Rust**: `Contract { symbol, sec_type: SecType, expiration:
    Option<i32>, is_call: Option<bool>, strike: Option<i32> }` plus
    `right() -> Option<Right>` (typed `Call` / `Put`) and
    `strike_dollars() -> Option<f64>`.
  - **Python**: `symbol: str`, `sec_type: str` (`"STOCK"` /
    `"OPTION"` / `"INDEX"` / `"RATE"`), `expiration: Optional[int]`,
    `right: Optional[str]` (`"C"` / `"P"`), `strike_dollars:
    Optional[float]`, `strike: Optional[int]` (wire form).
  - **TypeScript**: same field set, camelCase (`secType`,
    `strikeDollars`).
  - **C / C++**: `TdxContract { symbol; sec_type; has_expiration;
    expiration; has_right; right; has_strike; strike; }`. ASCII
    `'C'` / `'P'` for `right`; integer wire form for `strike`. C++
    inline helpers `tdx::strike_dollars`, `tdx::right`,
    `tdx::sec_type_name`, `tdx::reason_name` mirror the Python / TS
    fluent fields.
- **Hard rename** — `ThetaDataDx` → `ThetaDataDxClient`. No alias.

# Typed control variants

Every `FpssControl::*` Rust variant has a dedicated typed surface:

- **Python**: one pyclass per variant (`LoginSuccess`,
  `ContractAssigned`, `Disconnected`, `Reconnecting`, `Reconnected`,
  `MarketOpen`, `MarketClose`, `ServerError`, `Error`,
  `UnknownFrame`, `UnknownControl`, `Connected`, `Ping`,
  `ReconnectedServer`, `Restart`, `ReqResponse`). Branch on
  `event.kind` (snake_case), read the variant's typed payload
  directly. `Disconnected.reason_name` / `Reconnecting.reason_name`
  surface the `RemoveReason` enum name.
- **TypeScript**: one `#[napi(object)]` struct per variant;
  `event.disconnected.reasonName` mirrors the Python surface.
- **C / C++**: one `typedef struct { ... } TdxFpss<Variant>;` per
  variant. Dispatch on `event->kind` (`TdxFpssEventKind` enum), read
  the matching `event-><variant>` payload.

`FpssControl::ContractAssigned` keeps its diagnostic role; the
resolved typed `Arc<Contract>` is carried directly on every data
event, so user code never has to thread a contract-id side table.

# FLATFILES SDK parity

Hand-written FLATFILES bindings on Python, TypeScript, and C++ —
every endpoint the Rust crate exposes is reachable. Dynamic
per-(sec_type, req_type) schema converted to Arrow under the
`arrow` feature; Python returns `pyarrow.Table`, TypeScript returns
Arrow IPC bytes, C++ returns `TdxFlatFileRowList` with
`to_arrow_ipc()`. MCP server ships `tdx_flatfile_*` tool surfaces
matching the SDK shape.

# Cross-language utility parity

`condition_name(code) -> str`, `exchange_name(code) -> str`,
`sequence_signed_to_unsigned` / `sequence_unsigned_to_signed`
exposed across Python, TypeScript, and C++ — same names, same
return types. TypeScript helpers reject out-of-range BigInt inputs
at the i32 / u32 wire-range boundary instead of silently coercing.

# Encoding crate (`tdbe`)

- `mdds::decode::v3` renamed to `mdds::decode::dual_type_columns` to
  reflect what the module decodes, not which schema version it was
  written for. Schema bumped to v5.
- `Error` enum folded `Decode` / `Decompress` / `Config` / `Grpc`
  payloads into typed kinds; ~120 callsites migrated.
- `RemoveReason::from_code(i16) -> Self` and
  `RemoveReason::as_str(&self) -> &'static str` for the symbolic
  name accessors used by the Python / TS bindings.
- Canonical Gregorian calendar validator across MDDS, FPSS, and
  flat-files.

# Repository / CI

- Root tree trimmed; ADRs inlined into Rust doc comments; generated
  SDK files moved under `_generated/` subdirectories.
- TypeScript `npm test` runs all 6 test files (19 tests pass) on
  every advertised platform (macOS, Linux, Windows × py3.9 + py3.14).
- Stale `RawData` / `Empty` event variants hidden from the public
  surface; `contract_id` integer removed from all data events in
  favour of the typed `Arc<Contract>`.
- Observability path surfaces `SystemTime` and JSON-serialization
  failures instead of swallowing them.

# Breaking changes since v9.0.x

| Surface | Before | After |
|---|---|---|
| Client type name | `ThetaDataDx` | `ThetaDataDxClient` |
| Subscribe API | `client.subscribe_quotes(symbol)` / `subscribe_full_trades(sec_type)` and the per-kind family | `client.subscribe(Contract::stock("AAPL").quote())` / `client.subscribe(SecType::Option.full_trades())` |
| Option builder | `Contract::option(symbol, occ_string)` (2-arg) | `Contract::option(symbol, expiration, strike, right)` (4-arg, all strings) |
| C ABI Contract option side | `bool has_is_call; bool is_call;` | `bool has_right; char right;` (ASCII `'C'` / `'P'`) |
| Python / TS event-Contract | `sec_type: int`, `is_call: Optional[bool]` | `sec_type: str`, `right: Optional[str]`, plus `strike_dollars` |
| Control envelope | flat `TdxFpssControl { kind, id, detail }` | per-variant typed structs (`TdxFpssLoginSuccess`, `TdxFpssDisconnected`, …) |
| Data event contract id | `contract_id: i32` plus side-table lookup | typed `Arc<Contract>` directly on the event |
| Hidden internals | `FpssData::RawData` / `FpssData::Empty` public | hidden from user surface |
| Go SDK | shipped | removed end-to-end |

`cargo semver-checks --baseline-rev v9.0.0` reports the breaking
set; the release lands as a 9.x minor on the v9 line.

# Closes

- #513 streaming SSOT
- #424 cross-language utility parity
- #431, #432, #433, #441, #442, #446 FLATFILES ecosystem completion
- #471 tdbe condition flag (held for v9.2.0)
- #512 v3.rs rename
@userFRM userFRM merged commit d2cfe63 into main May 9, 2026
74 of 78 checks passed
@userFRM userFRM deleted the fix/h1-streaming-ssot branch May 9, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant