Skip to content

Commit 5444d80

Browse files
committed
fix(v9.1.0): close codex r12 audit findings — sequence helper validation, more doc drift, ROADMAP wave vocabulary
Three findings from the r12 PR-wide audit on PR #514, all closed. 1. **TS sequence helpers reject out-of-range BigInt (Medium)** — the prior pass relied on `BigInt::get_u64()` which conflates negative inputs with truly out-of-range bigints, so `sequenceUnsignedToSigned(-1n)` silently returned `1n` instead of throwing, and `sequenceSignedToUnsigned(2^63)` saturated to `i64::MAX` instead of throwing. Replaced with a hand-written `bigint_to_i32` helper that walks the napi `(sign_bit, words)` representation directly. The helpers now: - clamp the signed input to the i32 wire range (`-2_147_483_648..=2_147_483_647`) — this is the actual terminal-protocol wire range; the SDK widens to i64 internally but the round-trip is only meaningful in i32 (per `crates/tdbe/src/sequences.rs:8-14`); - clamp the unsigned input to the unsigned wire range (`0..=2^32 - 1`); - throw on negative BigInt to `sequenceUnsignedToSigned`; - accept the asymmetric `i32::MIN` boundary correctly. Test suite gained range / rejection coverage; `npm test` runs 19 / 19. 2. **Doc drift remainder (Medium)** — purged the stale public-surface references the prior pass missed: - `docs/api-reference.md`: TypeScript section described streaming as `tdx.startStreaming()` / `tdx.nextEvent()`. Both removed; the section now points to push-callback (`tdx.startStreaming(callback)`) and pull-iter (`for await (const event of await tdx.startStreamingIter())`). The TypeScript code example was fully rewritten against the modern API (`Contract.stock(...).quote()`, `await client.startStreamingIter()`, `awaitDrain` on shutdown). The contract-fields table dropped the Go-only column. - `docs-site/docs/streaming/connection.md`: replaced the `nextEvent()` row in the async/sync table with the actual `EventIterator.next(timeoutMs)` shape. - `docs-site/docs/getting-started/streaming.md`: the architecture paragraph dropped Go from the "polling queue" enumeration (Wave H removed Go end-to-end). 3. **ROADMAP.md vocabulary scrub (Low)** — `docs/ROADMAP.md` is a user-facing doc page and still leaked Wave / issue-tracker vocabulary (`Wave O`, `#431` / `#432` / `#433` / `#436` / `#438`, `#424`). The "Tracking" column on the binding-status table and the per-row `(Wave O)` annotations on the coverage matrix are gone; the open-work paragraph now says "Shipped" without the issue-number reference. Pre-push pipeline: cargo fmt --all -- --check, clippy --workspace --locked -- -D warnings, test --workspace --locked, deny check all, docs consistency, generate_sdk_surfaces --check, npm run build, npm test (19/19 pass), C++ CMake all green.
1 parent 7797759 commit 5444d80

7 files changed

Lines changed: 164 additions & 66 deletions

File tree

docs-site/docs/getting-started/streaming.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ graph LR
1818
C -->|"Disruptor SPSC<br/>131,072 slots"| D["Your app<br/>(callback / poll)"]
1919
```
2020

21-
Events are decoded from the FIT wire format and delta-decompressed on a dedicated I/O thread, then dispatched through an LMAX Disruptor SPSC ring buffer to your callback (Rust) or polling queue (Python / TypeScript / Go / C++). Every data event carries a `received_at_ns` nanosecond timestamp captured at frame decode time.
21+
Events are decoded from the FIT wire format and delta-decompressed on a dedicated I/O thread, then dispatched through an LMAX Disruptor SPSC ring buffer to your callback (push mode) or pull-iter consumer (Python / TypeScript / C++). Every data event carries a `received_at_ns` nanosecond timestamp captured at frame decode time.
2222

2323
## SPKI pinning
2424

docs-site/docs/streaming/connection.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -215,7 +215,7 @@ ThetaDataDx uses two different concurrency models for its two data paths:
215215
|------|---------|-----|
216216
| `connect()` + all historical methods | **async** (tokio) | gRPC/tonic requires tokio for HTTP/2 multiplexing |
217217
| `start_streaming()` + callbacks | **sync** (OS threads) | Dedicated I/O thread + LMAX Disruptor ring buffer for lowest latency |
218-
| TypeScript `nextEvent()` | **Promise** (napi-rs) | Returns `Promise<FpssEvent \| null>` so Node's event loop stays unblocked during the 50ms read timeout |
218+
| TypeScript `EventIterator.next(timeoutMs)` | **Promise** (napi-rs) | Returns `Promise<FpssEvent \| null>` (`null` = timeout / drained) so Node's event loop stays unblocked during the read timeout |
219219

220220
**What this means for your code:**
221221

docs/ROADMAP.md

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -133,31 +133,31 @@ Server retention window: 7 calendar days. Older history: contact ThetaData sales
133133

134134
### FLATFILES — Binding Coverage
135135

136-
| Binding | Status | Tracking |
137-
|---------|--------|----------|
138-
| Rust (`crates/thetadatadx`) | Shipped | v8.0.17 |
139-
| FFI (`ffi/`) | Shipped | #434 |
140-
| CLI (`tools/cli`) | Shipped | #433 (Wave O / v9.1.0) |
141-
| MCP (`tools/mcp`) | Shipped | #431 (Wave O / v9.1.0) |
142-
| REST/WS server (`tools/server`) | Shipped | #432 (Wave O / v9.1.0) |
143-
| Python (`sdks/python`) | Shipped | #435 |
144-
| TypeScript (`sdks/typescript`) | Shipped | #436 |
145-
| C++ (`sdks/cpp`) | Shipped | #438 |
146-
| `sdk_surface.toml` declarative spec | Skipped (hand-written) | #439 — flatfile bindings are intentionally hand-written; the dynamic `(SecType, ReqType)` schema doesn't fit the static codegen surface and the function set is finite. |
136+
| Binding | Status |
137+
|---------|--------|
138+
| Rust (`crates/thetadatadx`) | Shipped |
139+
| FFI (`ffi/`) | Shipped |
140+
| CLI (`tools/cli`) | Shipped |
141+
| MCP (`tools/mcp`) | Shipped |
142+
| REST/WS server (`tools/server`) | Shipped |
143+
| Python (`sdks/python`) | Shipped |
144+
| TypeScript (`sdks/typescript`) | Shipped |
145+
| C++ (`sdks/cpp`) | Shipped |
146+
| `sdk_surface.toml` declarative spec | Skipped (hand-written) — flatfile bindings are intentionally hand-written; the dynamic `(SecType, ReqType)` schema doesn't fit the static codegen surface and the function set is finite. |
147147

148148
## Binding Coverage Matrix
149149

150-
End-to-end status of each public surface across every SDK / tool surface ThetaDataDx ships. Wave O (v9.1.0) closes the FLATFILES gaps in the tools row and the cross-language utility helpers gap.
150+
End-to-end status of each public surface across every SDK / tool surface ThetaDataDx ships.
151151

152152
| Surface | Rust | Python | TypeScript | C | C++ | MCP | CLI | REST/WS |
153153
|---|---|---|---|---|---|---|---|---|
154154
| FPSS streaming push (callback) |||||| ✅ (control events surface as ticks) || ✅ (WS broadcast) |
155155
| FPSS streaming pull (iter) |||||||||
156156
| MDDS historical endpoints |||||||||
157-
| FLATFILES whole-universe blobs ||||||(Wave O) |(Wave O) | (Wave O) |
158-
| Cross-language utils — conditions ||(Wave O) |(Wave O) |(Wave O) | (Wave O) ||||
159-
| Cross-language utils — exchange ||(Wave O) |(Wave O) |(Wave O) | (Wave O) ||||
160-
| Cross-language utils — sequences ||(Wave O) |(Wave O) |(Wave O) | (Wave O) ||||
157+
| FLATFILES whole-universe blobs |||||||||
158+
| Cross-language utils — conditions |||||||||
159+
| Cross-language utils — exchange |||||||||
160+
| Cross-language utils — sequences |||||||||
161161
| Greeks (offline calculator) |||||||||
162162
| Implied volatility (offline) |||||||||
163163
| Authentication / config / ping |||||||||
@@ -185,7 +185,7 @@ Empty cells (`—`) are intentional: the MCP / CLI / REST surfaces don't expose
185185

186186
### Cross-language parity for `utils`
187187

188-
Shipped in Wave O (v9.1.0). The Rust SDK exposes `tdbe::{conditions, exchange, sequences}` for post-processing tick records, and every public binding now mirrors that surface. Tracked under issue #424.
188+
Shipped. The Rust SDK exposes `tdbe::{conditions, exchange, sequences}` for post-processing tick records, and every public binding mirrors that surface.
189189

190190
- [x] **Python**`thetadatadx.util.{condition_name, exchange_name, exchange_symbol, sequence_signed_to_unsigned, ...}` via PyO3.
191191
- [x] **TypeScript**`Util.{conditionName, exchangeName, exchangeSymbol, sequenceSignedToUnsigned, ...}` via napi-rs.

docs/api-reference.md

Lines changed: 24 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -699,7 +699,7 @@ Every historical endpoint is available in the Python SDK via PyO3 bindings (e.g.
699699

700700
### TypeScript/Node.js SDK Coverage
701701

702-
Every historical endpoint is available in the TypeScript/Node.js SDK via napi-rs bindings as camelCase methods (e.g., `tdx.stockHistoryEOD(...)`). Streaming is available via `tdx.startStreaming()` / `tdx.nextEvent()`. Returns columnar objects with typed fields.
702+
Every historical endpoint is available in the TypeScript/Node.js SDK via napi-rs bindings as camelCase methods (e.g., `tdx.stockHistoryEOD(...)`). Streaming is available in two modes: push-callback (`tdx.startStreaming(callback)`) and pull-iter (`for await (const event of await tdx.startStreamingIter())`); both return typed objects with the same field shape. Returns columnar objects with typed fields.
703703

704704
### Python SDK: Streaming
705705

@@ -902,22 +902,29 @@ Check `event->kind` then read the corresponding field. Only the field matching `
902902
### TypeScript/Node.js SDK: Streaming
903903

904904
```typescript
905-
import { ThetaDataDxClient } from 'thetadatadx';
905+
import { ThetaDataDxClient, Contract } from 'thetadatadx';
906906

907907
const tdx = await ThetaDataDxClient.connectFromFile('creds.txt');
908908

909-
tdx.startStreaming();
910-
tdx.subscribe(ContractRef.stock('AAPL').quote());
911-
while (true) {
912-
const event = tdx.nextEvent(5000);
913-
if (!event) continue;
914-
if (event.kind === 'quote') {
915-
console.log(`Quote: bid=${event.bid} ask=${event.ask}`);
916-
} else if (event.kind === 'trade') {
917-
console.log(`Trade: price=${event.price} size=${event.size}`);
909+
tdx.subscribe(Contract.stock('AAPL').quote());
910+
911+
// Pull-iter mode: async iterable over the SPSC queue. Resolves
912+
// `done: true` once stopStreaming() fires and the queue drains.
913+
const iter = await tdx.startStreamingIter();
914+
try {
915+
for await (const event of iter) {
916+
if (event.kind === 'quote') {
917+
console.log(`Quote: bid=${event.bid} ask=${event.ask}`);
918+
} else if (event.kind === 'trade') {
919+
console.log(`Trade: price=${event.price} size=${event.size}`);
920+
} else if (event.kind === 'disconnected') {
921+
break;
922+
}
918923
}
924+
} finally {
925+
tdx.stopStreaming();
926+
await tdx.awaitDrain(5000);
919927
}
920-
tdx.stopStreaming();
921928
```
922929

923930
### C++ SDK: Streaming
@@ -1082,11 +1089,11 @@ All generated tick types are `Clone + Debug` structs generated from `tick_schema
10821089

10831090
> **Note:** Only `expiration` and `strike` support wildcards (`"0"`). The `right` parameter does **not** accept wildcards -- you must specify `"C"` or `"P"`.
10841091
1085-
| Field | Type (Rust/FFI) | Type (Go) | Description |
1086-
|-------|-----------------|-----------|-------------|
1087-
| `expiration` | `i32` | `int32` | Contract expiration date (YYYYMMDD). 0 on single-contract queries. |
1088-
| `strike` | `i32` | `int32` | Strike price (f64, decoded at parse time). |
1089-
| `right` | `i32` | `string` | Contract right. Rust/FFI: `67` = Call, `80` = Put, `0` = absent. Go: `"C"`, `"P"`, `""`. |
1092+
| Field | Type (Rust/FFI) | Description |
1093+
|-------|-----------------|-------------|
1094+
| `expiration` | `i32` | Contract expiration date (YYYYMMDD). 0 on single-contract queries. |
1095+
| `strike` | `i32` | Strike price (wire integer in thousandths of a dollar; `f64` after decode at parse time). |
1096+
| `right` | `i32` | Contract right. Rust/FFI ASCII byte: `67` (`'C'`) = Call, `80` (`'P'`) = Put, `0` = absent. Python / TypeScript surface this as `right: Optional[str]` (`"C"` / `"P"`). |
10901097

10911098
Helper methods on all 10 tick types:
10921099

sdks/typescript/__tests__/util_helpers.test.mjs

Lines changed: 40 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,45 @@ describe('Util cross-language helpers (#424)', () => {
3232
assert.equal(mod.Util.exchangeName(-1), 'UNKNOWN');
3333
assert.equal(mod.Util.exchangeSymbol(9999), 'UNKNOWN');
3434

35-
// Sequence helpers — bidirectional round-trip.
36-
const signed = -1n;
37-
const unsigned = mod.Util.sequenceSignedToUnsigned(signed);
38-
assert.equal(typeof unsigned, 'bigint');
39-
assert.equal(mod.Util.sequenceUnsignedToSigned(unsigned), signed);
35+
// Sequence helpers — bidirectional round-trip across the i32
36+
// wire range, including the asymmetric `i32::MIN` boundary. The
37+
// upstream Java terminal encodes trade sequences as i32; the SDK
38+
// widens to i64 internally, but the meaningful round-trip is the
39+
// i32 range.
40+
for (const signed of [-1n, 0n, 1n, 2147483647n, -2147483648n]) {
41+
const unsigned = mod.Util.sequenceSignedToUnsigned(signed);
42+
assert.equal(typeof unsigned, 'bigint');
43+
assert.equal(mod.Util.sequenceUnsignedToSigned(unsigned), signed);
44+
}
45+
});
46+
47+
it('rejects BigInt inputs outside the wire range instead of silent coercion', async () => {
48+
let mod;
49+
try {
50+
mod = await import('../index.js');
51+
} catch {
52+
console.log('SKIP: native addon not built');
53+
return;
54+
}
55+
// i32::MAX + 1 — out of wire range.
56+
assert.throws(
57+
() => mod.Util.sequenceSignedToUnsigned(2147483648n),
58+
/i32 wire range/,
59+
);
60+
// i32::MIN - 1 — out of wire range.
61+
assert.throws(
62+
() => mod.Util.sequenceSignedToUnsigned(-2147483649n),
63+
/i32 wire range/,
64+
);
65+
// Negative BigInt for the unsigned helper — never valid.
66+
assert.throws(
67+
() => mod.Util.sequenceUnsignedToSigned(-1n),
68+
/negative BigInt rejected/,
69+
);
70+
// Greater than 2^32 - 1 — out of unsigned wire range.
71+
assert.throws(
72+
() => mod.Util.sequenceUnsignedToSigned(4294967296n),
73+
/wire range/,
74+
);
4075
});
4176
});

sdks/typescript/index.d.ts

Lines changed: 14 additions & 5 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

sdks/typescript/src/util_helpers.rs

Lines changed: 67 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -75,31 +75,78 @@ impl Util {
7575

7676
/// Convert a signed wire-encoded trade-sequence value to its unsigned
7777
/// monotonic form. Mirrors `tdbe::sequences::signed_to_unsigned`.
78-
/// Accepts a JS BigInt for the full i64 wire range; returns a JS
79-
/// BigInt because the full u64 range cannot fit in a JS Number.
78+
/// Accepts a JS BigInt in the **i32 wire range**
79+
/// (`-2_147_483_648 ..= 2_147_483_647`) — the upstream Java
80+
/// terminal encodes trade sequences as i32; the SDK widens to
81+
/// i64 internally, but the meaningful round-trip is the i32
82+
/// range. Returns a JS BigInt because the unsigned monotonic
83+
/// sequence id can exceed `Number.MAX_SAFE_INTEGER`. Inputs
84+
/// outside the i32 wire range throw so silent coercion cannot
85+
/// produce a look-correct-but-wrong sequence id downstream.
8086
#[napi(js_name = "sequenceSignedToUnsigned")]
81-
pub fn sequence_signed_to_unsigned(signed_value: BigInt) -> BigInt {
82-
let (signed_neg, value, _) = signed_value.get_u64();
83-
// `BigInt::get_u64` returns the magnitude in `value` and the
84-
// sign in `signed_neg`. Reconstruct the i64 wire value by
85-
// negating when the sign bit is set; saturating because the
86-
// wire format itself is bounded by i64::MIN..=i64::MAX.
87-
let signed = if signed_neg {
88-
i64::try_from(value).map_or(i64::MIN, i64::wrapping_neg)
89-
} else {
90-
i64::try_from(value).unwrap_or(i64::MAX)
91-
};
92-
BigInt::from(tdbe::sequences::signed_to_unsigned(signed))
87+
pub fn sequence_signed_to_unsigned(signed_value: BigInt) -> napi::Result<BigInt> {
88+
let signed: i64 = bigint_to_i32(&signed_value).map(i64::from).ok_or_else(|| {
89+
napi::Error::from_reason(
90+
"sequenceSignedToUnsigned: BigInt outside the i32 wire range \
91+
(-2_147_483_648 ..= 2_147_483_647)",
92+
)
93+
})?;
94+
Ok(BigInt::from(tdbe::sequences::signed_to_unsigned(signed)))
9395
}
9496

9597
/// Convert an unsigned monotonic trade-sequence value back to its
9698
/// signed wire encoding. Mirrors `tdbe::sequences::unsigned_to_signed`.
97-
/// Accepts a JS BigInt for the full u64 input range; returns a
98-
/// JS BigInt because the i64 wire range exceeds
99-
/// `Number.MAX_SAFE_INTEGER`.
99+
/// Accepts a JS BigInt in the unsigned wire range
100+
/// (`0 ..= SEQUENCE_RANGE - 1`, i.e. `0 ..= 2^32 - 1`); returns a
101+
/// JS BigInt for symmetry with `sequenceSignedToUnsigned`.
102+
/// Negative inputs and inputs above the wire range throw — the
103+
/// unsigned monotonic sequence id is always non-negative and
104+
/// never wider than the i32 wire range.
100105
#[napi(js_name = "sequenceUnsignedToSigned")]
101-
pub fn sequence_unsigned_to_signed(unsigned_value: BigInt) -> BigInt {
102-
let (_, value, _) = unsigned_value.get_u64();
103-
BigInt::from(tdbe::sequences::unsigned_to_signed(value))
106+
pub fn sequence_unsigned_to_signed(unsigned_value: BigInt) -> napi::Result<BigInt> {
107+
if unsigned_value.sign_bit && !unsigned_value.words.iter().all(|w| *w == 0) {
108+
return Err(napi::Error::from_reason(
109+
"sequenceUnsignedToSigned: negative BigInt rejected; the unsigned \
110+
monotonic sequence id is always non-negative",
111+
));
112+
}
113+
if unsigned_value.words.len() > 1 {
114+
return Err(napi::Error::from_reason(
115+
"sequenceUnsignedToSigned: BigInt above the wire range \
116+
(0 ..= 2^32 - 1)",
117+
));
118+
}
119+
let value = unsigned_value.words.first().copied().unwrap_or(0);
120+
if value > u32::MAX as u64 {
121+
return Err(napi::Error::from_reason(
122+
"sequenceUnsignedToSigned: BigInt above the wire range \
123+
(0 ..= 2^32 - 1)",
124+
));
125+
}
126+
Ok(BigInt::from(tdbe::sequences::unsigned_to_signed(value)))
127+
}
128+
}
129+
130+
/// Decode a napi `BigInt` into the i32 wire range, accepting the
131+
/// asymmetric `i32::MIN` boundary. Returns `None` for any value
132+
/// outside `[i32::MIN, i32::MAX]`.
133+
fn bigint_to_i32(value: &BigInt) -> Option<i32> {
134+
if value.words.len() > 1 {
135+
return None;
136+
}
137+
let magnitude = value.words.first().copied().unwrap_or(0);
138+
if value.sign_bit {
139+
if magnitude == 0 {
140+
Some(0)
141+
} else if magnitude <= i32::MAX as u64 {
142+
// SAFETY: `magnitude` fits in i32 here.
143+
Some(-(magnitude as i32))
144+
} else if magnitude == (i32::MAX as u64) + 1 {
145+
Some(i32::MIN)
146+
} else {
147+
None
148+
}
149+
} else {
150+
i32::try_from(magnitude).ok()
104151
}
105152
}

0 commit comments

Comments
 (0)