docs(v9.1.0): scrub stale RawData / FLATFILES copy + schema v5

userFRM · claude · userFRM · commit 80a8fb168127 · 2026-05-07T21:43:53.000+02:00
- Strip references to the deleted public `FpssEvent::RawData` variant
  from every shipped surface (Python / TS / C++ READMEs, api-reference,
  architecture, the C header, and the C++ smoke example). Note that
  truncated / unrecognised wire frames are now filtered before the
  callback fires and accounted on the
  `thetadatadx.fpss.decode_failures` metric counter.
- Replace the "FLATFILES not yet exposed" coverage notes (Python / TS /
  C++ / FFI READMEs) with a description of the now-shipped
  `tdx.flat_files.*()` / `tdx.flatFiles.*()` /
  `unified.flat_files().*()` / C ABI surface and matching terminals
  (`.to_arrow()`, `.to_polars()`, `.to_pandas()`, `.to_arrow_ipc()`).
- Bump the schema-version drift in `.github/release-notes/v9.1.0.md`,
  `CHANGELOG.md`, and `docs-site/docs/changelog.md` from "version 4" to
  "version 5", matching `crates/thetadatadx/fpss_event_schema.toml`.

No public API change. Version stays 9.0.2.

Co-Authored-By: Claude Opus 4.7 (1M context) &lt;noreply@anthropic.com&gt;
diff --git a/.github/release-notes/v9.1.0.md b/.github/release-notes/v9.1.0.md
@@ -22,7 +22,7 @@ the user callback directly from the Disruptor consumer thread.
   `tdx::Fpss<Variant>` aliases. The `TdxFpssEventKind` enum gains
   one discriminant per control variant — symbolic names
   (`TDX_FPSS_LOGIN_SUCCESS`, ...) stay stable. Schema bumped to
-  version 4 (`crates/thetadatadx/fpss_event_schema.toml`); generated
+  version 5 (`crates/thetadatadx/fpss_event_schema.toml`); generated
   outputs regenerated; codegen idempotency check enforced in CI.
   Migration table for the old flat `control.kind` integer →
   new typed struct path: see `CHANGELOG.md` § Migration from v9.0.x.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -86,7 +86,7 @@ code does not change.
   gains one discriminant per control variant — numeric values
   renumber alphabetically; symbolic names
   (`TDX_FPSS_LOGIN_SUCCESS`, `FpssLoginSuccessEvent`, etc.) are
-  stable. Schema bumped to version 4
+  stable. Schema bumped to version 5
   (`crates/thetadatadx/fpss_event_schema.toml`); generated outputs
   regenerated; codegen idempotency check enforced in CI. See the
   v9.0.x → v9.1.0 migration table for the old→new field mapping.
diff --git a/crates/thetadatadx/build_support/sdk_surface/python.rs b/crates/thetadatadx/build_support/sdk_surface/python.rs
@@ -105,8 +105,10 @@ fn python_streaming_method(method: &MethodSpec) -> String {
                  `MarketOpen`, `MarketClose`, `ServerError`,\n\
                  `Disconnected`, `Reconnecting`, `Reconnected`, `Error`,\n\
                  `UnknownFrame`, `Connected`, `Ping`, `ReconnectedServer`,\n\
-                 `Restart`, `UnknownControl`, or `RawData`). Dispatch via\n\
-                 `match event: case Quote(): ...`.\n\
+                 `Restart`, `UnknownControl`). Dispatch via\n\
+                 `match event: case Quote(): ...`. Truncated / unrecognised\n\
+                 wire frames are filtered before the callback fires and\n\
+                 accounted on the `thetadatadx.fpss.decode_failures` metric.\n\
                  \n\
                  Events flow from the FPSS reader thread into the\n\
                  LMAX Disruptor ring (`Producer::try_publish`) and out\n\
diff --git a/docs-site/docs/changelog.md b/docs-site/docs/changelog.md
@@ -86,7 +86,7 @@ code does not change.
   gains one discriminant per control variant — numeric values
   renumber alphabetically; symbolic names
   (`TDX_FPSS_LOGIN_SUCCESS`, `FpssLoginSuccessEvent`, etc.) are
-  stable. Schema bumped to version 4
+  stable. Schema bumped to version 5
   (`crates/thetadatadx/fpss_event_schema.toml`); generated outputs
   regenerated; codegen idempotency check enforced in CI. See the
   v9.0.x → v9.1.0 migration table for the old→new field mapping.
diff --git a/docs/api-reference.md b/docs/api-reference.md
@@ -793,13 +793,16 @@ struct, and `thetadx.hpp` guards every field via
 
 ```c
 /* TdxFpssEventKind discriminants enumerate every data variant
- * (Quote / Trade / OpenInterest / Ohlcvc), every typed control
+ * (Quote / Trade / OpenInterest / Ohlcvc) and every typed control
  * variant (LoginSuccess / ContractAssigned / Disconnected /
  * Reconnecting / ServerError / Error / Ping / UnknownFrame /
  * MarketOpen / MarketClose / Connected / Reconnected /
- * ReconnectedServer / Restart / ReqResponse / UnknownControl), plus
- * the RawData fallback. Numeric values renumber alphabetically on
- * each major bump; reach for the symbolic names. */
+ * ReconnectedServer / Restart / ReqResponse / UnknownControl).
+ * Numeric values renumber alphabetically on each major bump; reach
+ * for the symbolic names. Truncated / unrecognised wire frames are
+ * filtered inside the Rust core (accounted on
+ * `thetadatadx.fpss.decode_failures`) and never surface through the C
+ * ABI. */
 
 typedef struct {
     TdxFpssEventKind kind;
@@ -825,8 +828,6 @@ typedef struct {
     TdxFpssServerError       server_error;
     TdxFpssUnknownControl    unknown_control;
     TdxFpssUnknownFrame      unknown_frame;
-    /* Raw-bytes fallback */
-    TdxFpssRawData      raw_data;
 } TdxFpssEvent;
 ```
 
diff --git a/docs/architecture.md b/docs/architecture.md
@@ -339,9 +339,8 @@ FPSS event dispatch uses a lock-free disruptor ring buffer (`disruptor-rs` v4),
 Events delivered through the ring buffer use a split enum:
 - **`FpssEvent::Data(FpssData)`** — market data: `Quote`, `Trade`, `OpenInterest`, `Ohlcvc`
 - **`FpssEvent::Control(FpssControl)`** — lifecycle: `LoginSuccess`, `ContractAssigned`, `ReqResponse`, `MarketOpen`, `MarketClose`, `ServerError`, `Disconnected`, `Error`
-- **`FpssEvent::RawData`** — unparsed frames
 
-This split enables callers to `match` on data events without touching lifecycle logic, and vice versa.
+This split enables callers to `match` on data events without touching lifecycle logic, and vice versa. Truncated / unrecognised wire frames are accounted on the `thetadatadx.fpss.decode_failures` metric counter (carried internally as `FpssEventInternal::Unparseable`) and filtered before reaching the public callback.
 
 ### OHLCVC-from-Trade Derivation
 
diff --git a/ffi/README.md b/ffi/README.md
@@ -4,7 +4,7 @@ C FFI layer for `thetadatadx` — exposes the Rust SDK as `extern "C"` functions
 
 Compiled as both `cdylib` (shared library) and `staticlib` (archive). Consumed by the C++ (RAII) and TypeScript/Node.js (napi-rs) SDKs, and available to any third-party C/C++/Go/etc. consumer that wants to roll their own wrapper against the `tdx_*` symbols.
 
-> **FLATFILES coverage:** the FFI layer currently exposes the MDDS (historical) and FPSS (streaming) surfaces only. The third surface — FLATFILES whole-universe daily blobs — is shipped in the Rust core (v8.0.17+) and is being added to the C ABI under issue [#434](https://github.com/userFRM/ThetaDataDx/issues/434). The C++ binding (#438) tracks this issue as its upstream blocker. See [`ROADMAP.md`](../ROADMAP.md#flatfiles--binding-coverage) for the per-binding status.
+> **Surface coverage:** the FFI layer exposes all three ThetaData surfaces — MDDS (historical), FPSS (streaming), and FLATFILES (whole-universe daily blobs). The flat-files entry points are `tdx_flatfile_request_decoded` (pull + decode into an opaque row-list), `tdx_flatfile_rows_to_arrow_ipc` (serialise to Arrow IPC bytes), `tdx_flatfile_request_to_path` (raw vendor bytes straight to disk), and the matching `_rowlist_free` / `_bytes_free` cleanup helpers.
 
 ## Building
 
diff --git a/sdks/cpp/README.md b/sdks/cpp/README.md
@@ -4,7 +4,7 @@ C++ SDK for ThetaData market data. Header-only RAII wrappers over the `thetadata
 
 Every call crosses the C ABI boundary into compiled Rust: gRPC communication, protobuf parsing, zstd decompression, and TCP streaming run inside the `thetadatadx` crate.
 
-> **FLATFILES coverage:** the C++ binding currently exposes the MDDS (historical) and FPSS (streaming) surfaces only. The third surface — FLATFILES whole-universe daily blobs — is shipped in the Rust core (v8.0.17+) and is being wired through the FFI layer (issue [#434](https://github.com/userFRM/ThetaDataDx/issues/434)) into C++ under issue [#438](https://github.com/userFRM/ThetaDataDx/issues/438). See [`ROADMAP.md`](../../ROADMAP.md#flatfiles--binding-coverage) for the per-binding status.
+> **Surface coverage:** the C++ binding exposes all three ThetaData surfaces — MDDS (historical), FPSS (streaming), and FLATFILES (whole-universe daily blobs). Flat files land via `unified.flat_files().*()` with `.to_arrow_ipc()` terminals plus a `flat_files().to_path(...)` raw-bytes helper — see the [Flat Files](#flat-files) section for the full method list.
 
 ## Prerequisites
 
@@ -352,7 +352,11 @@ All prices in streaming events are `double` (f64) -- decoded during parsing. Acc
 | `TdxFpssReconnectedServer` | (none) | `kind == TDX_FPSS_RECONNECTED_SERVER` |
 | `TdxFpssRestart` | (none) | `kind == TDX_FPSS_RESTART` |
 | `TdxFpssUnknownControl` | (none) | `kind == TDX_FPSS_UNKNOWN_CONTROL` |
-| `TdxFpssRawData` | code (u8), payload (`uint8_t*`), payload_len (size_t) | `kind == TDX_FPSS_RAW_DATA` |
+
+Truncated / unrecognised wire frames are filtered before the user
+callback fires and accounted on the `thetadatadx.fpss.decode_failures`
+metric counter on the Rust side; they never surface through the C ABI
+event stream.
 
 `FpssClient` is non-copyable but movable. The destructor calls `shutdown()` automatically.
 
diff --git a/sdks/cpp/examples/fpss_smoke.cpp b/sdks/cpp/examples/fpss_smoke.cpp
@@ -164,10 +164,6 @@ int main(int argc, char** argv) {
                     // Payload-less variants — discriminator alone carries
                     // the meaning.
                     break;
-                case TDX_FPSS_RAW_DATA:
-                    std::cout << " code=" << static_cast<int>(event.raw_data.code)
-                              << " len=" << event.raw_data.payload_len;
-                    break;
             }
             std::cout << std::endl;
         });
diff --git a/sdks/cpp/include/thetadx.h b/sdks/cpp/include/thetadx.h
@@ -562,9 +562,8 @@ int tdx_implied_volatility(double spot, double strike, double rate, double div_y
  *
  * Borrowed pointers (`Contract.symbol`, `LoginSuccess.permissions`,
  * `ServerError.message`, `Error.message`, `Ping.payload`,
- * `UnknownFrame.payload`, `RawData.payload`) are valid only for the
- * duration of the user callback — copy out before returning. Do NOT
- * free. */
+ * `UnknownFrame.payload`) are valid only for the duration of the
+ * user callback — copy out before returning. Do NOT free. */
 #include "fpss_event_structs.h.inc"
 
 /* ═══════════════════════════════════════════════════════════════════════ */
diff --git a/sdks/python/README.md b/sdks/python/README.md
@@ -2,7 +2,7 @@
 
 Python bindings over the Rust core. Every call crosses the PyO3 boundary into Rust: gRPC communication, protobuf parsing, zstd decompression, FIT tick decoding, and TCP streaming run inside the `thetadatadx` crate.
 
-> **FLATFILES coverage:** the Python binding currently exposes the MDDS (historical) and FPSS (streaming) surfaces only. The third surface — FLATFILES whole-universe daily blobs — is shipped in the Rust core (v8.0.17+) and is being wired into Python under issue [#435](https://github.com/userFRM/ThetaDataDx/issues/435). See [`ROADMAP.md`](../../ROADMAP.md#flatfiles--binding-coverage) for the per-binding status.
+> **Surface coverage:** the Python binding exposes all three ThetaData surfaces — MDDS (historical), FPSS (streaming), and FLATFILES (whole-universe daily blobs). Flat files land via `tdx.flat_files.*()` with `.to_arrow()`, `.to_polars()`, `.to_pandas()`, and `.to_list()` terminals plus a `flatfile_to_path(...)` raw-bytes helper — see the [Flat Files](#flat-files) section for the full method list.
 
 ## Installation
 
@@ -236,10 +236,13 @@ Events arrive as typed pyclass instances — `Quote`, `Trade`, `Ohlcvc`,
 (`LoginSuccess`, `ContractAssigned`, `ReqResponse`, `MarketOpen`,
 `MarketClose`, `ServerError`, `Disconnected`, `Reconnecting`,
 `Reconnected`, `Error`, `UnknownFrame`, `Connected`, `Ping`,
-`ReconnectedServer`, `Restart`, `UnknownControl`); `RawData` for
-unrecognised wire frames. Each class mirrors its `FpssControl::*` /
-`FpssData::*` Rust variant one-for-one, so dispatch is a structural
-`match` — exactly the same shape Rust consumers use:
+`ReconnectedServer`, `Restart`, `UnknownControl`). Each class mirrors
+its `FpssControl::*` / `FpssData::*` Rust variant one-for-one, so
+dispatch is a structural `match` — exactly the same shape Rust
+consumers use. Truncated / unrecognised wire frames are filtered before
+the callback fires and accounted on the
+`thetadatadx.fpss.decode_failures` metric counter; they never surface
+on the public event stream.
 
 ```python
 from thetadatadx import (
@@ -377,7 +380,7 @@ the blocking `get()`.
 | `contract_lookup(id)` | Look up a single contract by ID (returns str or None) |
 | `active_subscriptions()` | Get list of active subscriptions (list of dicts with "kind" and "contract") |
 | `streaming(callback)` | Open a context-managed streaming session. `with tdx.streaming(callback) as session:` registers `callback` via `start_streaming` on enter and pairs `stop_streaming()` + `await_drain(5_000)` on exit, mirroring the C++ RAII destructor. Subscription methods on the bound `session` proxy through to the underlying `ThetaDataDx` via `StreamingSession.__getattr__` -- single source of truth. |
-| `start_streaming(callback)` | Register a callable; the LMAX Disruptor consumer thread invokes `callback(event)` under the GIL for every typed FPSS event. `event` is a `Quote` / `Trade` / `Ohlcvc` / `OpenInterest` for market data, one of the typed control classes (`LoginSuccess`, `ContractAssigned`, `Disconnected`, `Reconnecting`, ...) for lifecycle events, or `RawData` for unrecognised frames — dispatch via Python `match`. Each invocation is wrapped in `catch_unwind`. `event.kind` carries the same discriminator tag as the TypeScript SDK's `FpssEvent.kind`. |
+| `start_streaming(callback)` | Register a callable; the LMAX Disruptor consumer thread invokes `callback(event)` under the GIL for every typed FPSS event. `event` is a `Quote` / `Trade` / `Ohlcvc` / `OpenInterest` for market data or one of the typed control classes (`LoginSuccess`, `ContractAssigned`, `Disconnected`, `Reconnecting`, ...) for lifecycle events — dispatch via Python `match`. Truncated / unrecognised wire frames are filtered before the callback fires (counted on `thetadatadx.fpss.decode_failures`). Each invocation is wrapped in `catch_unwind`. `event.kind` carries the same discriminator tag as the TypeScript SDK's `FpssEvent.kind`. |
 | `await_drain(timeout_ms)` | Block until the previous streaming session's consumer thread has finished firing the registered callback. Returns `True` if the drain completed within `timeout_ms`, `False` otherwise. Use after `stop_streaming()` or `reconnect()` from a thread other than the callback thread to confirm the callback closure can be safely released. The `with tdx.streaming(...)` context manager calls this for you with a 5_000 ms timeout. |
 | `dropped_event_count()` | Cumulative count of events the FPSS reader could not publish into the Disruptor ring because the consumer fell behind and the ring was full. Resets to 0 on `reconnect()` (which rebuilds the FPSS client) and reads 0 after `stop_streaming()`. Snapshot the value before reconnect if you need to accumulate drops across session boundaries. |
 | `reconnect()` | Reconnect streaming and re-register the previously installed callback; restores all subscriptions. |
diff --git a/sdks/python/src/streaming_methods.rs b/sdks/python/src/streaming_methods.rs
diff --git a/sdks/typescript/README.md b/sdks/typescript/README.md
@@ -4,7 +4,7 @@ Node.js SDK for ThetaData market data. napi-rs bindings over the `thetadatadx` R
 
 Every call crosses the napi boundary into compiled Rust: gRPC, protobuf, zstd, FIT decoding, and TCP streaming run inside the `thetadatadx` crate.
 
-> **FLATFILES coverage:** the TypeScript binding currently exposes the MDDS (historical) and FPSS (streaming) surfaces only. The third surface — FLATFILES whole-universe daily blobs — is shipped in the Rust core (v8.0.17+) and is being wired into TypeScript under issue [#436](https://github.com/userFRM/ThetaDataDx/issues/436). See [`ROADMAP.md`](../../ROADMAP.md#flatfiles--binding-coverage) for the per-binding status.
+> **Surface coverage:** the TypeScript binding exposes all three ThetaData surfaces — MDDS (historical), FPSS (streaming), and FLATFILES (whole-universe daily blobs). Flat files land via `tdx.flatFiles.*()` with `.toArrowIpc()` and `.toJson()` terminals plus a `tdx.flatFileToPath(...)` raw-bytes helper — see the [Flat Files](#flat-files) section for the full method list.
 
 ## Install
 
@@ -127,13 +127,14 @@ tdx.startStreaming((event: FpssEvent) => {
     case 'reconnected_server':  /* event.reconnectedServer is ReconnectedServer */    break;
     case 'restart':             /* event.restart is Restart */                        break;
     case 'unknown_control':     /* event.unknownControl is UnknownControl */          break;
-
-    // Unrecognised wire frames
-    case 'raw_data':            /* event.rawData is FpssRawDataPayload */             break;
   }
 });
 ```
 
+Truncated / unrecognised wire frames are filtered before the callback
+fires and accounted on the `thetadatadx.fpss.decode_failures` metric
+counter on the Rust side; they never surface as an `FpssEvent`.
+
 The `kind` field is typed as a string-literal union narrowed by the
 generated `index.d.ts` — plain strings, not a TS `enum` (the previous
 `const enum FpssEventKind` was removed in #376 because it broke
