From 0942e90cd12eae94016974942272d5ec11eff8c7 Mon Sep 17 00:00:00 2001 From: preview Date: Wed, 17 Jun 2026 22:36:42 +0200 Subject: [PATCH] feat(python): add awaitable flat-file terminals for event-loop-safe AsyncClient use The flat-file namespace was reachable from AsyncClient through the unified proxy allowlist, but its only terminals were synchronous. A flat-file pull is a full-day blob download of seconds, so reaching option_trade_quote(date) and friends through AsyncClient.flat_files drove the whole download to completion on the event loop thread and stalled every other coroutine for its duration, defeating the non-blocking guarantee AsyncClient is built to provide. There was no awaitable flat-file terminal on the async surface. Add awaitable *_async twins for every flat-file fetch on FlatFilesNamespace (option_trade_quote_async, option_open_interest_async, option_eod_async, stock_trade_quote_async, stock_eod_async, request_async) plus Client.flatfile_to_path_async, each resolving the download off the event loop through the same awaitable bridge the historical *_async terminals use. The synchronous methods keep their blocking behaviour for plain Client use, so await flat_files.option_eod_async(date) inside a coroutine yields the same FlatFileRowList without holding the loop. The bridge never holds the interpreter lock across the download; the row materialisation runs only after the data has arrived. Declare the new awaitable methods in the .pyi stub with Awaitable return types, verified against the built module with stubtest. The flat-file reverse-orphan parity scan now recognises an _async member as the awaitable twin of its enrolled sync fetch by matching the base name, mirroring how the cross-binding async surface is tracked elsewhere; an _async method whose base is not an enrolled fetch still trips the gate, so the check is not weakened. Co-Authored-By: Claude Opus 4.8 --- CHANGELOG.md | 1 + docs-site/docs/changelog.md | 1 + scripts/check_binding_parity.py | 15 ++- sdks/python/python/thetadatadx/__init__.pyi | 50 ++++++++ sdks/python/src/flatfile_methods.rs | 122 ++++++++++++++++++++ 5 files changed, 187 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ae636f58b..4f9122137 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - An Arrow-IPC terminal on the TypeScript and C++ history results — per-collection `ToArrowIpc(rows)` (TypeScript) and `thetadatadx::_to_arrow_ipc(rows)` (C++) emit the same Arrow IPC stream bytes as the existing flat-file terminal; an empty result is a valid zero-row stream carrying the schema. - A `from_file` client-construction convenience across the bindings: the Python unified `Client.from_file(path, config=None)`, the C-ABI `thetadatadx_*_connect_from_file(path, config)` trio, and the C++ `from_file(path, config = Config::production())` statics, all defaulting to the production configuration so a credentials file is the only required input. - Python `AsyncClient` gains awaitable constructors `await AsyncClient.connect(creds, config)` and `await AsyncClient.connect_from_file(path, config=None)` so async callers can establish a connection from inside a coroutine without the authentication handshake stalling the running event loop. The synchronous `AsyncClient(creds, config)` and `AsyncClient.from_file(...)` constructors stay available for construction outside a running loop. +- Python flat-file fetches gain awaitable `*_async` twins so the full-day blob download resolves off the event loop when reached through `AsyncClient.flat_files`: `option_trade_quote_async` / `option_open_interest_async` / `option_eod_async` / `stock_trade_quote_async` / `stock_eod_async` / `request_async` on the namespace yield the same `FlatFileRowList`, and `Client.flatfile_to_path_async(...)` yields the on-disk path. The synchronous methods keep their blocking behaviour for plain `Client` use; `await flat_files.option_eod_async(date)` inside a coroutine no longer stalls the running loop for the duration of the download. - TypeScript precomputed epoch-instant fields on every tick that carries a `date` plus a milliseconds-of-day column (`createdTimestampMs`, `lastTradeTimestampMs`, `timestampMs`, `underlyingTimestampMs`, `quoteTimestampMs`), one-for-one with the Python `*_timestamp_ms` properties and resolved through the same DST-aware core conversion. - TypeScript `Subscription` exposes the `contract` and `secType` getters Python already had, and `toString()` rendering is available on the TypeScript `ContractRef`, `Subscription`, and `SecType` values; C++ gains `operator<<` and a `thetadatadx::str(...)` rendering for the same fluent value types. - Trade flag-word accessors are generated into every binding (previously Rust-only on `TradeTick`): `is_cancelled`, `regular_trading_hours`, `is_seller`, `trade_condition_no_last`, `price_condition_set_last`, and `is_incremental_volume` (Python computed properties, TypeScript precomputed boolean fields, C++ free functions). The C ABI also gains `thetadatadx_contract_strike_dollars`, the dollar-valued counterpart of the existing C++ `thetadatadx::strike(...)` accessor. diff --git a/docs-site/docs/changelog.md b/docs-site/docs/changelog.md index ae636f58b..4f9122137 100644 --- a/docs-site/docs/changelog.md +++ b/docs-site/docs/changelog.md @@ -50,6 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - An Arrow-IPC terminal on the TypeScript and C++ history results — per-collection `ToArrowIpc(rows)` (TypeScript) and `thetadatadx::_to_arrow_ipc(rows)` (C++) emit the same Arrow IPC stream bytes as the existing flat-file terminal; an empty result is a valid zero-row stream carrying the schema. - A `from_file` client-construction convenience across the bindings: the Python unified `Client.from_file(path, config=None)`, the C-ABI `thetadatadx_*_connect_from_file(path, config)` trio, and the C++ `from_file(path, config = Config::production())` statics, all defaulting to the production configuration so a credentials file is the only required input. - Python `AsyncClient` gains awaitable constructors `await AsyncClient.connect(creds, config)` and `await AsyncClient.connect_from_file(path, config=None)` so async callers can establish a connection from inside a coroutine without the authentication handshake stalling the running event loop. The synchronous `AsyncClient(creds, config)` and `AsyncClient.from_file(...)` constructors stay available for construction outside a running loop. +- Python flat-file fetches gain awaitable `*_async` twins so the full-day blob download resolves off the event loop when reached through `AsyncClient.flat_files`: `option_trade_quote_async` / `option_open_interest_async` / `option_eod_async` / `stock_trade_quote_async` / `stock_eod_async` / `request_async` on the namespace yield the same `FlatFileRowList`, and `Client.flatfile_to_path_async(...)` yields the on-disk path. The synchronous methods keep their blocking behaviour for plain `Client` use; `await flat_files.option_eod_async(date)` inside a coroutine no longer stalls the running loop for the duration of the download. - TypeScript precomputed epoch-instant fields on every tick that carries a `date` plus a milliseconds-of-day column (`createdTimestampMs`, `lastTradeTimestampMs`, `timestampMs`, `underlyingTimestampMs`, `quoteTimestampMs`), one-for-one with the Python `*_timestamp_ms` properties and resolved through the same DST-aware core conversion. - TypeScript `Subscription` exposes the `contract` and `secType` getters Python already had, and `toString()` rendering is available on the TypeScript `ContractRef`, `Subscription`, and `SecType` values; C++ gains `operator<<` and a `thetadatadx::str(...)` rendering for the same fluent value types. - Trade flag-word accessors are generated into every binding (previously Rust-only on `TradeTick`): `is_cancelled`, `regular_trading_hours`, `is_seller`, `trade_condition_no_last`, `price_condition_set_last`, and `is_incremental_volume` (Python computed properties, TypeScript precomputed boolean fields, C++ free functions). The C ABI also gains `thetadatadx_contract_strike_dollars`, the dollar-valued counterpart of the existing C++ `thetadatadx::strike(...)` accessor. diff --git a/scripts/check_binding_parity.py b/scripts/check_binding_parity.py index d22ef87c1..96f6874b5 100644 --- a/scripts/check_binding_parity.py +++ b/scripts/check_binding_parity.py @@ -1286,6 +1286,7 @@ def _collect_cpp_class_methods(cpp_hpp: pathlib.Path) -> dict[str, set[str]]: { "handle_", "pull_decoded", + "pull_decoded_async", "thetadatadx_flatfile_request_decoded", "thetadatadx_flatfile_request_to_path", "to_path", @@ -1480,8 +1481,18 @@ def _check_method_rows( flatfiles_members |= ts_methods.get("FlatFilesNamespace", set()) flatfiles_members |= cpp_methods.get(_cpp_class_for("FlatFilesNamespace"), set()) for member in sorted(flatfiles_members - FLATFILES_NAMESPACE_EXEMPT_MEMBERS): - camel = _snake_to_camel(member) - if camel in enrolled_flatfiles_methods or member in enrolled_flatfiles_methods: + # A Python `_async` member is the awaitable twin of its sync + # fetch, not a distinct fetch contract: the TypeScript fetch methods + # are already `async` (Promise) and C++ exposes a `std::future` + # companion, so the async surface rides the same enrolled row rather + # than carrying its own. Strip a trailing `_async` and require the + # base to be an enrolled fetch (mirroring how the historical `_async` + # members are matched by their base endpoint). This does not weaken + # the scan: an `_async` member whose base is not an enrolled fetch + # still trips below. + base = member[: -len("_async")] if member.endswith("_async") else member + camel = _snake_to_camel(base) + if camel in enrolled_flatfiles_methods or base in enrolled_flatfiles_methods: continue errors.append( f" FlatFilesNamespace.{member}: fetch method present on the " diff --git a/sdks/python/python/thetadatadx/__init__.pyi b/sdks/python/python/thetadatadx/__init__.pyi index 7f9db48c6..7b49b7edf 100644 --- a/sdks/python/python/thetadatadx/__init__.pyi +++ b/sdks/python/python/thetadatadx/__init__.pyi @@ -1276,6 +1276,21 @@ class Client: auto-appended if absent). """ ... + def flatfile_to_path_async( + self, + sec_type: str, + req_type: str, + date: str, + path: str, + format: Optional[str] = None, + ) -> Awaitable[str]: + """Awaitable twin of :py:meth:`flatfile_to_path`. + + Resolves the blob download off the calling thread so a running + event loop keeps servicing other coroutines while the file streams + to disk. Yields the final on-disk path. + """ + ... def __repr__(self) -> str: """Return a representation including historical and streaming state.""" @@ -1749,6 +1764,15 @@ class FlatFilesNamespace: Each method maps one ``(SecType, ReqType)`` pair to a :class:`FlatFileRowList`. The wildcard :py:meth:`request` dispatches dynamically by string identifiers. + + Every fetch carries an ``*_async`` twin returning an awaitable. A + flat-file pull is a full-day blob download that takes seconds; the + plain methods run that to completion on the calling thread, which is + right for a :class:`Client` call but would stall a running event loop + when reached through :py:attr:`AsyncClient.flat_files`. Inside a + coroutine, ``await flat_files.option_eod_async(date)`` resolves the + download without blocking the loop and yields the same + :class:`FlatFileRowList`. """ def option_trade_quote(self, date: str) -> FlatFileRowList: @@ -1788,6 +1812,32 @@ class FlatFilesNamespace: """ ... + def option_trade_quote_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable option-trade-quote flat file for ``date`` (``YYYYMMDD``).""" + ... + + def option_open_interest_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable option-open-interest flat file for ``date`` (``YYYYMMDD``).""" + ... + + def option_eod_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable option-EOD flat file for ``date`` (``YYYYMMDD``).""" + ... + + def stock_trade_quote_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable stock-trade-quote flat file for ``date`` (``YYYYMMDD``).""" + ... + + def stock_eod_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable stock-EOD flat file for ``date`` (``YYYYMMDD``).""" + ... + + def request_async( + self, sec_type: str, req_type: str, date: str + ) -> Awaitable[FlatFileRowList]: + """Awaitable twin of :py:meth:`request`, resolved off the event loop.""" + ... + class ThetaDataError(Exception): """Base exception for every typed error this binding raises.""" diff --git a/sdks/python/src/flatfile_methods.rs b/sdks/python/src/flatfile_methods.rs index 869d4ab29..5bcac73db 100644 --- a/sdks/python/src/flatfile_methods.rs +++ b/sdks/python/src/flatfile_methods.rs @@ -30,6 +30,7 @@ use pyo3::types::PyList; use thetadatadx::flatfiles::{self, FlatFileFormat, FlatFileRow, FlatFileValue, ReqType, SecType}; +use crate::async_runtime::spawn_awaitable; use crate::errors::to_py_err; use crate::record_batch_to_pyarrow_table; use crate::run_blocking; @@ -200,6 +201,22 @@ impl FlatFilesNamespace { })?; Ok(FlatFileRowList { rows }) } + + fn pull_decoded_async<'py>( + &self, + py: Python<'py>, + sec: SecType, + req: ReqType, + date: &str, + ) -> PyResult> { + let client = Arc::clone(&self.client); + let date_owned = date.to_string(); + spawn_awaitable( + py, + async move { client.flatfile_request_decoded(sec, req, &date_owned).await }, + |py, rows| Ok(Py::new(py, FlatFileRowList { rows })?.into_any()), + ) + } } #[pymethods] @@ -248,6 +265,71 @@ impl FlatFilesNamespace { let req = parse_flatfile_req_type(req_type)?; self.pull_decoded(py, sec, req, date) } + + // ── Awaitable terminals ───────────────────────────────────────── + // + // Each `*_async` method is the awaitable twin of the sync method + // above. A flat-file pull is a full-day blob download — seconds of + // network plus a decode pass. The sync methods drive that to + // completion on the calling thread, which is correct for a plain + // `Client` call but would stall a running asyncio event loop when + // reached through `AsyncClient.flat_files`. The awaitable terminals + // resolve the download off the event loop so other coroutines keep + // running while the day's data arrives, then yield the same + // `FlatFileRowList`. Pick the suffix that matches your call site: + // sync `flat_files.option_eod(date)` on a `Client`, awaitable + // `await flat_files.option_eod_async(date)` inside a coroutine. + + /// Awaitable option-trade-quote flat file for `date` (YYYYMMDD). + fn option_trade_quote_async<'py>( + &self, + py: Python<'py>, + date: &str, + ) -> PyResult> { + self.pull_decoded_async(py, SecType::Option, ReqType::TradeQuote, date) + } + + /// Awaitable option-open-interest flat file for `date` (YYYYMMDD). + fn option_open_interest_async<'py>( + &self, + py: Python<'py>, + date: &str, + ) -> PyResult> { + self.pull_decoded_async(py, SecType::Option, ReqType::OpenInterest, date) + } + + /// Awaitable option-EOD flat file for `date` (YYYYMMDD). + fn option_eod_async<'py>(&self, py: Python<'py>, date: &str) -> PyResult> { + self.pull_decoded_async(py, SecType::Option, ReqType::Eod, date) + } + + /// Awaitable stock-trade-quote flat file for `date` (YYYYMMDD). + fn stock_trade_quote_async<'py>( + &self, + py: Python<'py>, + date: &str, + ) -> PyResult> { + self.pull_decoded_async(py, SecType::Stock, ReqType::TradeQuote, date) + } + + /// Awaitable stock-EOD flat file for `date` (YYYYMMDD). + fn stock_eod_async<'py>(&self, py: Python<'py>, date: &str) -> PyResult> { + self.pull_decoded_async(py, SecType::Stock, ReqType::Eod, date) + } + + /// Awaitable generic dispatcher. `sec_type` and `req_type` accept the + /// same strings as `request(...)`, e.g. `"OPTION"` / `"QUOTE"`. + fn request_async<'py>( + &self, + py: Python<'py>, + sec_type: &str, + req_type: &str, + date: &str, + ) -> PyResult> { + let sec = parse_flatfile_sec_type(sec_type)?; + let req = parse_flatfile_req_type(req_type)?; + self.pull_decoded_async(py, sec, req, date) + } } // ── Client pymethods extension ──────────────────────────────────── @@ -305,4 +387,44 @@ impl crate::Client { })?; Ok(final_path.to_string_lossy().into_owned()) } + + /// Awaitable twin of [`flatfile_to_path`](Self::flatfile_to_path). + /// + /// Writes the requested format straight to `path` without decoding + /// into rows, resolving the blob download off the calling thread so a + /// running event loop keeps servicing other coroutines while the + /// day's file streams to disk. Yields the final on-disk path. + #[pyo3(signature = (sec_type, req_type, date, path, format=None))] + fn flatfile_to_path_async<'py>( + &self, + py: Python<'py>, + sec_type: &str, + req_type: &str, + date: &str, + path: &str, + format: Option<&str>, + ) -> PyResult> { + let sec = parse_flatfile_sec_type(sec_type)?; + let req = parse_flatfile_req_type(req_type)?; + let fmt = parse_flatfile_format(format)?; + let client = Arc::clone(&self.client); + let date_owned = date.to_string(); + let path_owned = std::path::PathBuf::from(path); + spawn_awaitable( + py, + async move { + client + .flatfile_request(sec, req, &date_owned, &path_owned, fmt) + .await + }, + |py, final_path| { + Ok(final_path + .to_string_lossy() + .into_owned() + .into_pyobject(py)? + .into_any() + .unbind()) + }, + ) + } }