feat(v9.1.0)!: streaming SSOT + fluent contract-first API + typed control variants + FLATFILES SDK parity#515
Merged
Merged
Conversation
…trol variants + FLATFILES SDK parity
Single-queue SSOT for the FPSS streaming pipeline, polymorphic
contract-first subscription API, typed `FpssControl::*` variants on
every binding, hand-written FLATFILES surface across Python /
TypeScript / C++, dynamic-schema Arrow conversion, and resilience
hardening across the FPSS lifecycle. The Go SDK is dropped; Python,
TypeScript, C, and C++ are first-class.
This is a v9.1.0 minor release with breaking API changes; the v9.0
line had no external consumers, so the release rolls forward as a
9.x minor instead of bumping to 10.
# Streaming pipeline
- One queue. The user callback runs as the LMAX Disruptor consumer's
`handle_events_with` closure under `std::panic::catch_unwind`. The
legacy dispatcher shim and `crossbeam-channel` runtime dep are gone.
- **Pull-iter delivery** — `start_streaming_iter()` (Rust / Python /
C++), `streaming_iter()` Python context manager, `for await` async
iterator (TypeScript), `TdxFpssEventIterator` (C / C++). Mutually
exclusive with the push-callback path on the same client; the
iterator drains the SPSC queue from the user thread.
- **Drain barrier** — `await_drain(timeout)` /
`tdx_*_await_drain` quiescence; `tdx_*_free` polls it (5 s) before
destroying the handle, closing the `ctx` use-after-free window.
- **Stop / restart** — `stop_streaming()` resurrection race closed
with a generation token; multi-generation drain via
`prev_drained: Mutex<Vec<Arc<AtomicBool>>>` walks every prior
generation's flag.
- **Reconnect** — `tdx_unified_reconnect` and `tdx_fpss_reconnect`
block on the previous-generation drain flag (5 s budget) before
binding the new session to the same C callback / `ctx`. The
documented "single-thread callback" contract is now actually
enforced by the FFI.
- **Self-join detach** — `FpssClient::Drop` detects the
consumer-thread self-join case via `OnceLock<ThreadId>` and
detaches `io_handle.join()` onto a helper thread.
- **Config knobs** — `FpssConfig` exposes `ring_size`, `flush_mode`,
`tcp_nodelay`, `tcp_keepalive`, `connect_timeout`, `read_timeout`
on the client builder. TypeScript shape-manifest agreement gap
closed.
# API fluency on every binding
- **Subscriptions** — one polymorphic `subscribe(spec)` /
`unsubscribe(spec)` method on the unified `ThetaDataDxClient`. Build
a typed spec via `quote()` / `trade()` / `open_interest()` on
`Contract`, or `full_trades()` / `full_open_interest()` on
`SecType`. `subscribe_many` / `unsubscribe_many` for bulk. The
legacy per-kind family is gone.
- **Contract builder** — `Contract::stock("AAPL")`,
`Contract::index("SPX")`, `Contract::option(symbol, expiration,
strike, right)` (all strings positional in C++ / Rust; keyword-only
in Python). All four arguments required.
- **Event-carried Contract is fluent** across languages — `event.contract`
exposes:
- **Rust**: `Contract { symbol, sec_type: SecType, expiration:
Option<i32>, is_call: Option<bool>, strike: Option<i32> }` plus
`right() -> Option<Right>` (typed `Call` / `Put`) and
`strike_dollars() -> Option<f64>`.
- **Python**: `symbol: str`, `sec_type: str` (`"STOCK"` /
`"OPTION"` / `"INDEX"` / `"RATE"`), `expiration: Optional[int]`,
`right: Optional[str]` (`"C"` / `"P"`), `strike_dollars:
Optional[float]`, `strike: Optional[int]` (wire form).
- **TypeScript**: same field set, camelCase (`secType`,
`strikeDollars`).
- **C / C++**: `TdxContract { symbol; sec_type; has_expiration;
expiration; has_right; right; has_strike; strike; }`. ASCII
`'C'` / `'P'` for `right`; integer wire form for `strike`. C++
inline helpers `tdx::strike_dollars`, `tdx::right`,
`tdx::sec_type_name`, `tdx::reason_name` mirror the Python / TS
fluent fields.
- **Hard rename** — `ThetaDataDx` → `ThetaDataDxClient`. No alias.
# Typed control variants
Every `FpssControl::*` Rust variant has a dedicated typed surface:
- **Python**: one pyclass per variant (`LoginSuccess`,
`ContractAssigned`, `Disconnected`, `Reconnecting`, `Reconnected`,
`MarketOpen`, `MarketClose`, `ServerError`, `Error`,
`UnknownFrame`, `UnknownControl`, `Connected`, `Ping`,
`ReconnectedServer`, `Restart`, `ReqResponse`). Branch on
`event.kind` (snake_case), read the variant's typed payload
directly. `Disconnected.reason_name` / `Reconnecting.reason_name`
surface the `RemoveReason` enum name.
- **TypeScript**: one `#[napi(object)]` struct per variant;
`event.disconnected.reasonName` mirrors the Python surface.
- **C / C++**: one `typedef struct { ... } TdxFpss<Variant>;` per
variant. Dispatch on `event->kind` (`TdxFpssEventKind` enum), read
the matching `event-><variant>` payload.
`FpssControl::ContractAssigned` keeps its diagnostic role; the
resolved typed `Arc<Contract>` is carried directly on every data
event, so user code never has to thread a contract-id side table.
# FLATFILES SDK parity
Hand-written FLATFILES bindings on Python, TypeScript, and C++ —
every endpoint the Rust crate exposes is reachable. Dynamic
per-(sec_type, req_type) schema converted to Arrow under the
`arrow` feature; Python returns `pyarrow.Table`, TypeScript returns
Arrow IPC bytes, C++ returns `TdxFlatFileRowList` with
`to_arrow_ipc()`. MCP server ships `tdx_flatfile_*` tool surfaces
matching the SDK shape.
# Cross-language utility parity
`condition_name(code) -> str`, `exchange_name(code) -> str`,
`sequence_signed_to_unsigned` / `sequence_unsigned_to_signed`
exposed across Python, TypeScript, and C++ — same names, same
return types. TypeScript helpers reject out-of-range BigInt inputs
at the i32 / u32 wire-range boundary instead of silently coercing.
# Encoding crate (`tdbe`)
- `mdds::decode::v3` renamed to `mdds::decode::dual_type_columns` to
reflect what the module decodes, not which schema version it was
written for. Schema bumped to v5.
- `Error` enum folded `Decode` / `Decompress` / `Config` / `Grpc`
payloads into typed kinds; ~120 callsites migrated.
- `RemoveReason::from_code(i16) -> Self` and
`RemoveReason::as_str(&self) -> &'static str` for the symbolic
name accessors used by the Python / TS bindings.
- Canonical Gregorian calendar validator across MDDS, FPSS, and
flat-files.
# Repository / CI
- Root tree trimmed; ADRs inlined into Rust doc comments; generated
SDK files moved under `_generated/` subdirectories.
- TypeScript `npm test` runs all 6 test files (19 tests pass) on
every advertised platform (macOS, Linux, Windows × py3.9 + py3.14).
- Stale `RawData` / `Empty` event variants hidden from the public
surface; `contract_id` integer removed from all data events in
favour of the typed `Arc<Contract>`.
- Observability path surfaces `SystemTime` and JSON-serialization
failures instead of swallowing them.
# Breaking changes since v9.0.x
| Surface | Before | After |
|---|---|---|
| Client type name | `ThetaDataDx` | `ThetaDataDxClient` |
| Subscribe API | `client.subscribe_quotes(symbol)` / `subscribe_full_trades(sec_type)` and the per-kind family | `client.subscribe(Contract::stock("AAPL").quote())` / `client.subscribe(SecType::Option.full_trades())` |
| Option builder | `Contract::option(symbol, occ_string)` (2-arg) | `Contract::option(symbol, expiration, strike, right)` (4-arg, all strings) |
| C ABI Contract option side | `bool has_is_call; bool is_call;` | `bool has_right; char right;` (ASCII `'C'` / `'P'`) |
| Python / TS event-Contract | `sec_type: int`, `is_call: Optional[bool]` | `sec_type: str`, `right: Optional[str]`, plus `strike_dollars` |
| Control envelope | flat `TdxFpssControl { kind, id, detail }` | per-variant typed structs (`TdxFpssLoginSuccess`, `TdxFpssDisconnected`, …) |
| Data event contract id | `contract_id: i32` plus side-table lookup | typed `Arc<Contract>` directly on the event |
| Hidden internals | `FpssData::RawData` / `FpssData::Empty` public | hidden from user surface |
| Go SDK | shipped | removed end-to-end |
`cargo semver-checks --baseline-rev v9.0.0` reports the breaking
set; the release lands as a 9.x minor on the v9 line.
# Closes
- #513 streaming SSOT
- #424 cross-language utility parity
- #431, #432, #433, #441, #442, #446 FLATFILES ecosystem completion
- #471 tdbe condition flag (held for v9.2.0)
- #512 v3.rs rename
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v9.1.0 single-queue SSOT for the FPSS streaming pipeline, polymorphic contract-first subscription API, typed
FpssControl::*variants on every binding, hand-written FLATFILES surface across Python / TypeScript / C++, dynamic-schema Arrow conversion, and resilience hardening across the FPSS lifecycle. The Go SDK is dropped; Python, TypeScript, C, and C++ are first-class.Streaming pipeline
handle_events_withclosure understd::panic::catch_unwind. The legacy dispatcher shim andcrossbeam-channelruntime dep are gone.start_streaming_iter()(Rust / Python / C++),streaming_iter()Python context manager,for awaitasync iterator (TypeScript),TdxFpssEventIterator(C / C++).await_drain(timeout)/tdx_*_await_drainquiescence;tdx_*_freepolls it (5 s) before destroying the handle.stop_streaming()resurrection race; multi-generation drain viaprev_drained: Mutex<Vec<Arc<AtomicBool>>>.tdx_unified_reconnectandtdx_fpss_reconnectblock on the previous-generation drain flag (5 s budget) before binding the new session to the same C callback /ctx.FpssClient::Dropdetects the consumer-thread self-join case viaOnceLock<ThreadId>and detachesio_handle.join()onto a helper thread.FpssConfigexposesring_size,flush_mode,tcp_nodelay,tcp_keepalive,connect_timeout,read_timeouton the client builder.API fluency
subscribe(spec)/unsubscribe(spec)method on the unifiedThetaDataDxClient. Build a typed spec viaquote()/trade()/open_interest()onContract, orfull_trades()/full_open_interest()onSecType.subscribe_many/unsubscribe_manyfor bulk.Contract::stock("AAPL"),Contract::index("SPX"),Contract::option(symbol, expiration, strike, right). All strings positional in C++ / Rust; keyword-only in Python.event.contractexposesright,strike_dollars,sec_type(symbolic name), and per-binding-idiomatic shapes for the option side. C++ inline helperstdx::strike_dollars,tdx::right,tdx::sec_type_name,tdx::reason_namemirror the Python / TS fluent fields.Typed control variants
Every
FpssControl::*Rust variant has a dedicated typed surface:LoginSuccess,ContractAssigned,Disconnected,Reconnecting, ...). Branch onevent.kind(snake_case), read the variant's typed payload directly.Disconnected.reason_name/Reconnecting.reason_namesurface theRemoveReasonenum name.#[napi(object)]struct per variant;event.disconnected.reasonNamemirrors the Python surface.typedef struct { ... } TdxFpss<Variant>;per variant. Dispatch onevent->kind(TdxFpssEventKindenum), read the matchingevent-><variant>payload.FpssControl::ContractAssignedkeeps its diagnostic role; the resolved typedArc<Contract>is carried directly on every data event, so user code never has to thread a contract-id side table.FLATFILES SDK parity
Hand-written FLATFILES bindings on Python, TypeScript, and C++ — every endpoint the Rust crate exposes is reachable. Dynamic per-(sec_type, req_type) schema converted to Arrow under the
arrowfeature; Python returnspyarrow.Table, TypeScript returns Arrow IPC bytes, C++ returnsTdxFlatFileRowListwithto_arrow_ipc(). MCP server shipstdx_flatfile_*tool surfaces matching the SDK shape.Cross-language utility parity
condition_name,exchange_name,sequence_signed_to_unsigned,sequence_unsigned_to_signedexposed across Python, TypeScript, and C++ — same names, same return types. TypeScript helpers reject out-of-range BigInt inputs at the i32 / u32 wire-range boundary instead of silently coercing.Encoding crate (
tdbe)mdds::decode::v3renamed tomdds::decode::dual_type_columns. Schema bumped to v5.Errorenum foldedDecode/Decompress/Config/Grpcpayloads into typed kinds; ~120 callsites migrated.RemoveReason::from_code(i16) -> SelfandRemoveReason::as_str(&self) -> &'static strfor the symbolic-name accessors used by Python / TS bindings.Repository / CI
_generated/subdirectories.npm testruns all 6 test files (19 tests pass) on every advertised platform.RawData/Emptyevent variants hidden from the public surface;contract_idinteger removed from all data events.SystemTimeand JSON-serialization failures instead of swallowing them.Breaking changes since v9.0.x
ThetaDataDxThetaDataDxClientclient.subscribe_quotes(symbol)/subscribe_full_trades(sec_type)and the per-kind familyclient.subscribe(Contract::stock("AAPL").quote())/client.subscribe(SecType::Option.full_trades())Contract::option(symbol, occ_string)(2-arg)Contract::option(symbol, expiration, strike, right)(4-arg, all strings)bool has_is_call; bool is_call;bool has_right; char right;(ASCII'C'/'P')sec_type: int,is_call: Optional[bool]sec_type: str,right: Optional[str], plusstrike_dollarsTdxFpssControl { kind, id, detail }TdxFpssLoginSuccess,TdxFpssDisconnected, ...)contract_id: i32plus side-table lookupArc<Contract>directly on the eventFpssData::RawData/FpssData::Emptypubliccargo semver-checks --baseline-rev v9.0.0reports the breaking set; the release lands as a 9.x minor on the v9 line.Closes
Test plan
cargo fmt --all -- --checkcargo clippy --workspace --locked -- -D warningscargo test --workspace --lockedcargo deny check allcargo build --release -p thetadatadx-ffi --lockednpm run build+npm test(19/19 pass)cargo check+ ABI3 smoke on macOS / Linux / Windows × py3.9, py3.14python3 scripts/check_docs_consistency.pyexits 0cargo run -p thetadatadx --bin generate_sdk_surfaces -- --checkreports no driftcargo semver-checks --baseline-rev v9.0.0— fails with the documented intentional-break set, not blocking