diff --git a/CHANGELOG.md b/CHANGELOG.md index ae636f58b..4f9122137 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -50,6 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - An Arrow-IPC terminal on the TypeScript and C++ history results — per-collection `ToArrowIpc(rows)` (TypeScript) and `thetadatadx::_to_arrow_ipc(rows)` (C++) emit the same Arrow IPC stream bytes as the existing flat-file terminal; an empty result is a valid zero-row stream carrying the schema. - A `from_file` client-construction convenience across the bindings: the Python unified `Client.from_file(path, config=None)`, the C-ABI `thetadatadx_*_connect_from_file(path, config)` trio, and the C++ `from_file(path, config = Config::production())` statics, all defaulting to the production configuration so a credentials file is the only required input. - Python `AsyncClient` gains awaitable constructors `await AsyncClient.connect(creds, config)` and `await AsyncClient.connect_from_file(path, config=None)` so async callers can establish a connection from inside a coroutine without the authentication handshake stalling the running event loop. The synchronous `AsyncClient(creds, config)` and `AsyncClient.from_file(...)` constructors stay available for construction outside a running loop. +- Python flat-file fetches gain awaitable `*_async` twins so the full-day blob download resolves off the event loop when reached through `AsyncClient.flat_files`: `option_trade_quote_async` / `option_open_interest_async` / `option_eod_async` / `stock_trade_quote_async` / `stock_eod_async` / `request_async` on the namespace yield the same `FlatFileRowList`, and `Client.flatfile_to_path_async(...)` yields the on-disk path. The synchronous methods keep their blocking behaviour for plain `Client` use; `await flat_files.option_eod_async(date)` inside a coroutine no longer stalls the running loop for the duration of the download. - TypeScript precomputed epoch-instant fields on every tick that carries a `date` plus a milliseconds-of-day column (`createdTimestampMs`, `lastTradeTimestampMs`, `timestampMs`, `underlyingTimestampMs`, `quoteTimestampMs`), one-for-one with the Python `*_timestamp_ms` properties and resolved through the same DST-aware core conversion. - TypeScript `Subscription` exposes the `contract` and `secType` getters Python already had, and `toString()` rendering is available on the TypeScript `ContractRef`, `Subscription`, and `SecType` values; C++ gains `operator<<` and a `thetadatadx::str(...)` rendering for the same fluent value types. - Trade flag-word accessors are generated into every binding (previously Rust-only on `TradeTick`): `is_cancelled`, `regular_trading_hours`, `is_seller`, `trade_condition_no_last`, `price_condition_set_last`, and `is_incremental_volume` (Python computed properties, TypeScript precomputed boolean fields, C++ free functions). The C ABI also gains `thetadatadx_contract_strike_dollars`, the dollar-valued counterpart of the existing C++ `thetadatadx::strike(...)` accessor. diff --git a/docs-site/docs/changelog.md b/docs-site/docs/changelog.md index ae636f58b..4f9122137 100644 --- a/docs-site/docs/changelog.md +++ b/docs-site/docs/changelog.md @@ -50,6 +50,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - An Arrow-IPC terminal on the TypeScript and C++ history results — per-collection `ToArrowIpc(rows)` (TypeScript) and `thetadatadx::_to_arrow_ipc(rows)` (C++) emit the same Arrow IPC stream bytes as the existing flat-file terminal; an empty result is a valid zero-row stream carrying the schema. - A `from_file` client-construction convenience across the bindings: the Python unified `Client.from_file(path, config=None)`, the C-ABI `thetadatadx_*_connect_from_file(path, config)` trio, and the C++ `from_file(path, config = Config::production())` statics, all defaulting to the production configuration so a credentials file is the only required input. - Python `AsyncClient` gains awaitable constructors `await AsyncClient.connect(creds, config)` and `await AsyncClient.connect_from_file(path, config=None)` so async callers can establish a connection from inside a coroutine without the authentication handshake stalling the running event loop. The synchronous `AsyncClient(creds, config)` and `AsyncClient.from_file(...)` constructors stay available for construction outside a running loop. +- Python flat-file fetches gain awaitable `*_async` twins so the full-day blob download resolves off the event loop when reached through `AsyncClient.flat_files`: `option_trade_quote_async` / `option_open_interest_async` / `option_eod_async` / `stock_trade_quote_async` / `stock_eod_async` / `request_async` on the namespace yield the same `FlatFileRowList`, and `Client.flatfile_to_path_async(...)` yields the on-disk path. The synchronous methods keep their blocking behaviour for plain `Client` use; `await flat_files.option_eod_async(date)` inside a coroutine no longer stalls the running loop for the duration of the download. - TypeScript precomputed epoch-instant fields on every tick that carries a `date` plus a milliseconds-of-day column (`createdTimestampMs`, `lastTradeTimestampMs`, `timestampMs`, `underlyingTimestampMs`, `quoteTimestampMs`), one-for-one with the Python `*_timestamp_ms` properties and resolved through the same DST-aware core conversion. - TypeScript `Subscription` exposes the `contract` and `secType` getters Python already had, and `toString()` rendering is available on the TypeScript `ContractRef`, `Subscription`, and `SecType` values; C++ gains `operator<<` and a `thetadatadx::str(...)` rendering for the same fluent value types. - Trade flag-word accessors are generated into every binding (previously Rust-only on `TradeTick`): `is_cancelled`, `regular_trading_hours`, `is_seller`, `trade_condition_no_last`, `price_condition_set_last`, and `is_incremental_volume` (Python computed properties, TypeScript precomputed boolean fields, C++ free functions). The C ABI also gains `thetadatadx_contract_strike_dollars`, the dollar-valued counterpart of the existing C++ `thetadatadx::strike(...)` accessor. diff --git a/scripts/check_binding_parity.py b/scripts/check_binding_parity.py index d22ef87c1..96f6874b5 100644 --- a/scripts/check_binding_parity.py +++ b/scripts/check_binding_parity.py @@ -1286,6 +1286,7 @@ def _collect_cpp_class_methods(cpp_hpp: pathlib.Path) -> dict[str, set[str]]: { "handle_", "pull_decoded", + "pull_decoded_async", "thetadatadx_flatfile_request_decoded", "thetadatadx_flatfile_request_to_path", "to_path", @@ -1480,8 +1481,18 @@ def _check_method_rows( flatfiles_members |= ts_methods.get("FlatFilesNamespace", set()) flatfiles_members |= cpp_methods.get(_cpp_class_for("FlatFilesNamespace"), set()) for member in sorted(flatfiles_members - FLATFILES_NAMESPACE_EXEMPT_MEMBERS): - camel = _snake_to_camel(member) - if camel in enrolled_flatfiles_methods or member in enrolled_flatfiles_methods: + # A Python `_async` member is the awaitable twin of its sync + # fetch, not a distinct fetch contract: the TypeScript fetch methods + # are already `async` (Promise) and C++ exposes a `std::future` + # companion, so the async surface rides the same enrolled row rather + # than carrying its own. Strip a trailing `_async` and require the + # base to be an enrolled fetch (mirroring how the historical `_async` + # members are matched by their base endpoint). This does not weaken + # the scan: an `_async` member whose base is not an enrolled fetch + # still trips below. + base = member[: -len("_async")] if member.endswith("_async") else member + camel = _snake_to_camel(base) + if camel in enrolled_flatfiles_methods or base in enrolled_flatfiles_methods: continue errors.append( f" FlatFilesNamespace.{member}: fetch method present on the " diff --git a/sdks/python/python/thetadatadx/__init__.pyi b/sdks/python/python/thetadatadx/__init__.pyi index 7f9db48c6..7b49b7edf 100644 --- a/sdks/python/python/thetadatadx/__init__.pyi +++ b/sdks/python/python/thetadatadx/__init__.pyi @@ -1276,6 +1276,21 @@ class Client: auto-appended if absent). """ ... + def flatfile_to_path_async( + self, + sec_type: str, + req_type: str, + date: str, + path: str, + format: Optional[str] = None, + ) -> Awaitable[str]: + """Awaitable twin of :py:meth:`flatfile_to_path`. + + Resolves the blob download off the calling thread so a running + event loop keeps servicing other coroutines while the file streams + to disk. Yields the final on-disk path. + """ + ... def __repr__(self) -> str: """Return a representation including historical and streaming state.""" @@ -1749,6 +1764,15 @@ class FlatFilesNamespace: Each method maps one ``(SecType, ReqType)`` pair to a :class:`FlatFileRowList`. The wildcard :py:meth:`request` dispatches dynamically by string identifiers. + + Every fetch carries an ``*_async`` twin returning an awaitable. A + flat-file pull is a full-day blob download that takes seconds; the + plain methods run that to completion on the calling thread, which is + right for a :class:`Client` call but would stall a running event loop + when reached through :py:attr:`AsyncClient.flat_files`. Inside a + coroutine, ``await flat_files.option_eod_async(date)`` resolves the + download without blocking the loop and yields the same + :class:`FlatFileRowList`. """ def option_trade_quote(self, date: str) -> FlatFileRowList: @@ -1788,6 +1812,32 @@ class FlatFilesNamespace: """ ... + def option_trade_quote_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable option-trade-quote flat file for ``date`` (``YYYYMMDD``).""" + ... + + def option_open_interest_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable option-open-interest flat file for ``date`` (``YYYYMMDD``).""" + ... + + def option_eod_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable option-EOD flat file for ``date`` (``YYYYMMDD``).""" + ... + + def stock_trade_quote_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable stock-trade-quote flat file for ``date`` (``YYYYMMDD``).""" + ... + + def stock_eod_async(self, date: str) -> Awaitable[FlatFileRowList]: + """Awaitable stock-EOD flat file for ``date`` (``YYYYMMDD``).""" + ... + + def request_async( + self, sec_type: str, req_type: str, date: str + ) -> Awaitable[FlatFileRowList]: + """Awaitable twin of :py:meth:`request`, resolved off the event loop.""" + ... + class ThetaDataError(Exception): """Base exception for every typed error this binding raises.""" diff --git a/sdks/python/src/flatfile_methods.rs b/sdks/python/src/flatfile_methods.rs index 869d4ab29..5bcac73db 100644 --- a/sdks/python/src/flatfile_methods.rs +++ b/sdks/python/src/flatfile_methods.rs @@ -30,6 +30,7 @@ use pyo3::types::PyList; use thetadatadx::flatfiles::{self, FlatFileFormat, FlatFileRow, FlatFileValue, ReqType, SecType}; +use crate::async_runtime::spawn_awaitable; use crate::errors::to_py_err; use crate::record_batch_to_pyarrow_table; use crate::run_blocking; @@ -200,6 +201,22 @@ impl FlatFilesNamespace { })?; Ok(FlatFileRowList { rows }) } + + fn pull_decoded_async<'py>( + &self, + py: Python<'py>, + sec: SecType, + req: ReqType, + date: &str, + ) -> PyResult> { + let client = Arc::clone(&self.client); + let date_owned = date.to_string(); + spawn_awaitable( + py, + async move { client.flatfile_request_decoded(sec, req, &date_owned).await }, + |py, rows| Ok(Py::new(py, FlatFileRowList { rows })?.into_any()), + ) + } } #[pymethods] @@ -248,6 +265,71 @@ impl FlatFilesNamespace { let req = parse_flatfile_req_type(req_type)?; self.pull_decoded(py, sec, req, date) } + + // ── Awaitable terminals ───────────────────────────────────────── + // + // Each `*_async` method is the awaitable twin of the sync method + // above. A flat-file pull is a full-day blob download — seconds of + // network plus a decode pass. The sync methods drive that to + // completion on the calling thread, which is correct for a plain + // `Client` call but would stall a running asyncio event loop when + // reached through `AsyncClient.flat_files`. The awaitable terminals + // resolve the download off the event loop so other coroutines keep + // running while the day's data arrives, then yield the same + // `FlatFileRowList`. Pick the suffix that matches your call site: + // sync `flat_files.option_eod(date)` on a `Client`, awaitable + // `await flat_files.option_eod_async(date)` inside a coroutine. + + /// Awaitable option-trade-quote flat file for `date` (YYYYMMDD). + fn option_trade_quote_async<'py>( + &self, + py: Python<'py>, + date: &str, + ) -> PyResult> { + self.pull_decoded_async(py, SecType::Option, ReqType::TradeQuote, date) + } + + /// Awaitable option-open-interest flat file for `date` (YYYYMMDD). + fn option_open_interest_async<'py>( + &self, + py: Python<'py>, + date: &str, + ) -> PyResult> { + self.pull_decoded_async(py, SecType::Option, ReqType::OpenInterest, date) + } + + /// Awaitable option-EOD flat file for `date` (YYYYMMDD). + fn option_eod_async<'py>(&self, py: Python<'py>, date: &str) -> PyResult> { + self.pull_decoded_async(py, SecType::Option, ReqType::Eod, date) + } + + /// Awaitable stock-trade-quote flat file for `date` (YYYYMMDD). + fn stock_trade_quote_async<'py>( + &self, + py: Python<'py>, + date: &str, + ) -> PyResult> { + self.pull_decoded_async(py, SecType::Stock, ReqType::TradeQuote, date) + } + + /// Awaitable stock-EOD flat file for `date` (YYYYMMDD). + fn stock_eod_async<'py>(&self, py: Python<'py>, date: &str) -> PyResult> { + self.pull_decoded_async(py, SecType::Stock, ReqType::Eod, date) + } + + /// Awaitable generic dispatcher. `sec_type` and `req_type` accept the + /// same strings as `request(...)`, e.g. `"OPTION"` / `"QUOTE"`. + fn request_async<'py>( + &self, + py: Python<'py>, + sec_type: &str, + req_type: &str, + date: &str, + ) -> PyResult> { + let sec = parse_flatfile_sec_type(sec_type)?; + let req = parse_flatfile_req_type(req_type)?; + self.pull_decoded_async(py, sec, req, date) + } } // ── Client pymethods extension ──────────────────────────────────── @@ -305,4 +387,44 @@ impl crate::Client { })?; Ok(final_path.to_string_lossy().into_owned()) } + + /// Awaitable twin of [`flatfile_to_path`](Self::flatfile_to_path). + /// + /// Writes the requested format straight to `path` without decoding + /// into rows, resolving the blob download off the calling thread so a + /// running event loop keeps servicing other coroutines while the + /// day's file streams to disk. Yields the final on-disk path. + #[pyo3(signature = (sec_type, req_type, date, path, format=None))] + fn flatfile_to_path_async<'py>( + &self, + py: Python<'py>, + sec_type: &str, + req_type: &str, + date: &str, + path: &str, + format: Option<&str>, + ) -> PyResult> { + let sec = parse_flatfile_sec_type(sec_type)?; + let req = parse_flatfile_req_type(req_type)?; + let fmt = parse_flatfile_format(format)?; + let client = Arc::clone(&self.client); + let date_owned = date.to_string(); + let path_owned = std::path::PathBuf::from(path); + spawn_awaitable( + py, + async move { + client + .flatfile_request(sec, req, &date_owned, &path_owned, fmt) + .await + }, + |py, final_path| { + Ok(final_path + .to_string_lossy() + .into_owned() + .into_pyobject(py)? + .into_any() + .unbind()) + }, + ) + } }