diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 474d24c2d..6bdd8d543 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -19,7 +19,7 @@ Note: `protoc` is required even if you're not modifying `.proto` files, because git clone https://github.com/userFRM/ThetaDataDx.git cd ThetaDataDx -# Run the full workspace test suite +# Run the core workspace test suite cargo test --workspace # For integration tests against ThetaData servers, create creds.txt: @@ -44,16 +44,30 @@ cargo test --workspace # 4. FFI build (if modified) cargo build --release -p thetadatadx-ffi -# 5. Python SDK (if modified) -cd sdks/python && maturin develop && cd ../.. +# 5. Extended surfaces (if modified) +cargo clippy --manifest-path tools/mcp/Cargo.toml -- -D warnings +cargo test --manifest-path tools/mcp/Cargo.toml +cargo clippy --manifest-path tools/server/Cargo.toml -- -D warnings +cargo test --manifest-path tools/server/Cargo.toml +cargo clippy --manifest-path tools/cli/Cargo.toml -- -D warnings +cargo test --manifest-path tools/cli/Cargo.toml + +# 6. Language SDK smoke checks (if modified) +cargo check --manifest-path sdks/python/Cargo.toml +(cd sdks/go && LD_LIBRARY_PATH=../../target/release go test ./...) +c++ -std=c++17 -fsyntax-only -I sdks/cpp/include sdks/cpp/src/thetadx.cpp +cmake -S sdks/cpp -B build/cpp +cmake --build build/cpp --target thetadatadx_cpp ``` -One-liner for the common case: +One-liner for the common Rust-only case: ```bash cargo fmt --all -- --check && cargo clippy --workspace -- -D warnings && cargo test --workspace ``` +For full Linux parity with the current CI `surfaces` job, run the extended-surface and language-SDK checks above as well. + ## Commit Convention We follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/). @@ -116,47 +130,44 @@ feat(core)!: replace DirectClient with ThetaDataDx unified client ## How to Add a New Endpoint -> **Deep dive:** see [`docs/macro-guide.md`](docs/macro-guide.md) for a full -> explanation of the macro system, type tags, and FFI wrappers. +> **Deep dive:** see [`docs/macro-guide.md`](docs/macro-guide.md) for the +> internal macro system and generated builder model. + +The endpoint-facing source of truth is now split across: +- `crates/thetadatadx/proto/external.proto` for the wire contract +- `crates/thetadatadx/endpoint_surface.toml` for the normalized SDK surface +- `crates/thetadatadx/endpoint_schema.toml` for DataTable parser layouts -All 61 endpoint methods are generated by the `parsed_endpoint!` macro in `direct.rs`. -Tick type structs and parsers are generated from `endpoint_schema.toml` by `build.rs`. +The build expands that metadata into the registry, shared endpoint runtime, and +`DirectClient` declarations automatically. 1. **Update the proto** (if the endpoint uses a new message type) - - Edit `crates/thetadatadx/proto/v3_endpoints.proto` + - Update `crates/thetadatadx/proto/external.proto` - `cargo build` regenerates Rust types automatically -2. **Add the column schema** (if the response has a new layout) +2. **Add or update the endpoint surface** + - Add an entry to `crates/thetadatadx/endpoint_surface.toml` + - Reuse existing `param_groups` / `templates` where possible + - `cargo build` validates the declared surface against `external.proto` and generates the registry/runtime/direct surfaces + +3. **Add the column schema** (if the response has a new layout) - Add a `[types.YourTick]` block to `crates/thetadatadx/endpoint_schema.toml` - - `cargo build` generates the struct and parser + - `cargo build` generates the parser - See `docs/endpoint-schema.md` for the TOML format - Note: tick type structs, `Price`, enums, codecs, and Greeks live in `crates/tdbe/`. If you add a new tick type or modify existing types, edit `tdbe` first. -3. **Wire up the endpoint in `direct.rs`** - ```rust - parsed_endpoint! { - /// Doc comment. - fn stock_history_vwap(symbol: &str, start: &str, end: &str) -> Vec; - grpc: get_stock_history_vwap; - request: StockHistoryVwapRequest; - query: StockHistoryVwapParams { - root: symbol.to_string(), - start_date: start.to_string(), - end_date: end.to_string(), - }; - parse: decode::parse_vwap_ticks; - dates: start, end; - } - ``` - -4. **Expose in downstream SDKs** +4. **Review the generated direct/runtime surfaces** + - Most endpoint additions should not require hand-editing `direct.rs` + - Only change `build_support/endpoints.rs` or the macro layer if the new endpoint shape cannot be expressed by the existing surface spec + +5. **Expose in downstream SDKs** - FFI: add `extern "C"` function in `ffi/src/lib.rs` - Python: add PyO3 method in `sdks/python/src/lib.rs` - Go: add method in `sdks/go/client.go` - C++: add method in `sdks/cpp/include/thetadx.hpp` and `sdks/cpp/src/thetadx.cpp` -5. **Update CHANGELOG.md** under `[Unreleased]` +6. **Update CHANGELOG.md** under `[Unreleased]` See `crates/thetadatadx/proto/MAINTENANCE.md` for the full step-by-step guide. @@ -173,16 +184,19 @@ Every PR must include: - Updated `CHANGELOG.md` if user-facing - Updated documentation if any public API changed -## How to Update After a ThetaData Terminal Update +## How to Update After a ThetaData Protocol Update -When ThetaData releases a new terminal version: +When ThetaData ships a new official proto revision: -1. Refer to `docs/reverse-engineering.md` for methodology -2. Drop new proto files in `crates/thetadatadx/proto/` -3. Update `endpoint_schema.toml` if column schemas changed -4. Run the full test suite to verify backwards compatibility +1. Replace `crates/thetadatadx/proto/external.proto` +2. Update `endpoint_surface.toml` when the normalized SDK surface changes +3. Update `endpoint_schema.toml` if DataTable column layouts changed +4. Run the relevant checks from the sections above 5. See `crates/thetadatadx/proto/MAINTENANCE.md` for the detailed guide +`docs/reverse-engineering.md` is kept as historical context for how the project +was originally bootstrapped, not as the primary maintenance workflow. + ## Community Join the ThetaData Discord for questions and discussion: **[discord.thetadata.us](https://discord.thetadata.us/)** diff --git a/README.md b/README.md index 4d1b51a09..be03b04f5 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ No-JVM ThetaData Terminal - native Rust SDK for direct market data access. | [`tools/cli/`](tools/cli/) | `tdx` CLI - all 61 endpoints from the command line | | [`tools/mcp/`](tools/mcp/) | MCP server - gives LLMs access to 64 tools over JSON-RPC | | [`tools/server/`](tools/server/) | REST+WS server - drop-in replacement for the Java terminal | -| [`docs/`](docs/) | Architecture, API reference, JVM deviations, reverse-engineering guide | +| [`docs/`](docs/) | Architecture, API reference, JVM deviations, and historical reverse-engineering notes | | [`docs-site/`](docs-site/) | mdBook documentation site (deployed to GitHub Pages) | | [`notebooks/`](notebooks/) | 7 Jupyter notebooks (101-107) | | [`examples/`](examples/) | Example programs and test scripts | @@ -143,7 +143,7 @@ All endpoints return fully typed native structs in every language. Zero raw JSON | [API Reference](docs/api-reference.md) | All 61 methods, 14 tick types, configuration options | | [Architecture](docs/architecture.md) | System design, wire protocols, TOML codegen pipeline | | [JVM Deviations](docs/jvm-deviations.md) | Intentional differences from the Java terminal | -| [Reverse-Engineering Guide](docs/reverse-engineering.md) | How to decompile the terminal and extract proto definitions | +| [Reverse-Engineering Guide](docs/reverse-engineering.md) | Historical archive of the original reverse-engineering process before the official proto handoff | | [Endpoint Schema](docs/endpoint-schema.md) | TOML codegen format for adding new types/columns | | [Java Class Mapping](docs/java-class-mapping.md) | All 588 Java terminal classes enumerated | | [Proto Maintenance](crates/thetadatadx/proto/MAINTENANCE.md) | Guide for ThetaData engineers updating proto files | diff --git a/crates/thetadatadx/Cargo.toml b/crates/thetadatadx/Cargo.toml index addf44070..4bcaded32 100644 --- a/crates/thetadatadx/Cargo.toml +++ b/crates/thetadatadx/Cargo.toml @@ -12,7 +12,7 @@ license = "GPL-3.0-or-later" keywords = ["thetadata", "market-data", "tick-data", "zero-copy", "low-latency"] categories = ["finance", "api-bindings", "network-programming", "encoding"] readme = "../../README.md" -include = ["src/**/*", "proto/**/*", "build.rs", "Cargo.toml", "endpoint_schema.toml"] +include = ["src/**/*", "proto/**/*", "build.rs", "build_support/**/*", "Cargo.toml", "endpoint_schema.toml", "endpoint_surface.toml"] [package.metadata.docs.rs] all-features = true diff --git a/crates/thetadatadx/README.md b/crates/thetadatadx/README.md index b11a1e9d7..83205b04d 100644 --- a/crates/thetadatadx/README.md +++ b/crates/thetadatadx/README.md @@ -31,7 +31,7 @@ tdx.stop_streaming(); src/ lib.rs - public re-exports (ThetaDataDx, Credentials, DirectConfig, Error) unified.rs - ThetaDataDx: single entry point, lazy streaming - direct.rs - DirectClient: 61 gRPC endpoints via parsed_endpoint! macro + direct.rs - DirectClient macros plus generated endpoint declarations auth/ - Nexus API authentication, credential parsing fpss/ - FPSS streaming client (sync, LMAX Disruptor ring buffer) codec/ - FIT nibble encoder/decoder, delta compression @@ -39,20 +39,34 @@ src/ decode.rs - DataTable -> typed tick parsing (generated from TOML) types/ - Tick structs, Price, enums (generated from TOML) greeks.rs - 22 Black-Scholes Greeks + IV solver - registry.rs - Endpoint metadata (generated from proto at build time) + registry.rs - Endpoint metadata (generated from the endpoint surface spec) error.rs - Error enum proto/ - endpoints.proto - shared types (DataTable, ResponseData, Price) - v3_endpoints.proto - v3 service (BetaThetaTerminal, 60 RPCs) - MAINTENANCE.md - guide for ThetaData engineers + external.proto - canonical MDDS wire contract from ThetaData + MAINTENANCE.md - endpoint/proto maintenance guide endpoint_schema.toml - single source of truth for tick type definitions -build.rs - proto compilation + endpoint registry + TOML codegen +endpoint_surface.toml - explicit endpoint surface spec for registry/direct/runtime generation +build.rs - small build entrypoint +build_support/ - build-time generators for tick decoding and endpoint surfaces ``` ## TOML Codegen All 14 tick types and their DataTable parsers are generated at compile time from `endpoint_schema.toml`. Adding a new column is one line in the TOML. See [docs/endpoint-schema.md](../../docs/endpoint-schema.md). +## Endpoint Surface Spec + +Endpoint projections are generated from the checked-in `endpoint_surface.toml` +file, which defines the normalized endpoint surface: names, descriptions, +parameter semantics, REST paths, return kinds, projection call-shapes, reusable +parameter groups, and endpoint templates. Templates support inheritance via +`extends`, so the spec can model repeated endpoint families without copying the +same parameter blocks across every declaration. + +The build pipeline validates that surface spec against `proto/external.proto` +before generating the registry, shared endpoint runtime, and `DirectClient` +endpoint declarations. + ## Tick Types | Type | Fields | Use | diff --git a/crates/thetadatadx/build.rs b/crates/thetadatadx/build.rs index 938d26a59..d6b01448b 100644 --- a/crates/thetadatadx/build.rs +++ b/crates/thetadatadx/build.rs @@ -1,1757 +1,6 @@ -use std::collections::HashMap; -use std::fmt::Write as _; -use std::path::Path; - -use serde::Deserialize; +#[path = "build_support/mod.rs"] +mod build_support; fn main() -> Result<(), Box> { - tonic_prost_build::configure() - .build_server(false) - .compile_protos(&["proto/external.proto"], &["proto"])?; - - generate_endpoint_registry()?; - generate_endpoint_runtime()?; - generate_direct_endpoints()?; - generate_tick_types()?; - Ok(()) -} - -// ───────────────────────────────────────────────────────────────────────────── -// Code-generate tick structs + parsers from endpoint_schema.toml -// ───────────────────────────────────────────────────────────────────────────── - -#[derive(Debug, Deserialize)] -struct Schema { - types: HashMap, -} - -#[derive(Debug, Deserialize)] -struct TickTypeDef { - #[serde(rename = "doc")] - _doc: String, - #[serde(rename = "copy")] - _copy: bool, - #[serde(default, rename = "align")] - _align: Option, - parser: String, - #[serde(default)] - required: Vec, - #[serde(default)] - eod_style: bool, - #[serde(default)] - contract_id: bool, - columns: Vec, -} - -#[derive(Debug, Deserialize)] -struct ColumnDef { - name: String, - field: String, - r#type: String, -} - -fn generate_tick_types() -> Result<(), Box> { - let schema_path = "endpoint_schema.toml"; - let schema_str = std::fs::read_to_string(schema_path)?; - let schema: Schema = toml::from_str(&schema_str)?; - - let out_dir = std::env::var("OUT_DIR")?; - - // Collect type names and sort for deterministic output. - let mut type_names: Vec<&String> = schema.types.keys().collect(); - type_names.sort(); - - // ── Generate decode_generated.rs (parser functions) ────────────────────── - // Struct definitions now live in the `tdbe` crate (crates/tdbe/src/types/tick.rs). - // Only parser functions are generated here. - let mut parsers = String::new(); - parsers.push_str( - "// Auto-generated by build.rs from endpoint_schema.toml \u{2014} do not edit manually.\n\n", - ); - // Note: tick types are imported by decode.rs via `use tdbe::types::tick::*;` - // No additional imports needed in the generated code. - - for type_name in &type_names { - let def = &schema.types[*type_name]; - generate_parser(&mut parsers, type_name, def); - } - - let parsers_dest = Path::new(&out_dir).join("decode_generated.rs"); - std::fs::write(&parsers_dest, &parsers)?; - - // Rerun if schema changes - println!("cargo:rerun-if-changed={schema_path}"); - - Ok(()) -} - -/// Generate a single parser function. -/// -/// Price columns are decoded to `f64` during extraction using the wire -/// `price_type` (internally fetched, never exposed on the public struct). -// Reason: code generator -- the match dispatch over column types cannot be meaningfully split. -#[allow(clippy::too_many_lines)] -fn generate_parser(out: &mut String, type_name: &str, def: &TickTypeDef) { - let fn_name = &def.parser; - - writeln!( - out, - "pub fn {fn_name}(table: &crate::proto::DataTable) -> Vec<{type_name}> {{" - ) - .unwrap(); - - // If eod_style, emit the local eod_num and eod_price helpers inline. - if def.eod_style { - out.push_str(" // EOD rows may have Price-typed cells or plain Number cells.\n"); - out.push_str(" fn eod_num(row: &crate::proto::DataValueList, idx: usize) -> i32 {\n"); - out.push_str(" row.values\n"); - out.push_str(" .get(idx)\n"); - out.push_str(" .and_then(|dv| dv.data_type.as_ref())\n"); - out.push_str(" .and_then(|dt| match dt {\n"); - out.push_str( - " crate::proto::data_value::DataType::Number(n) => Some(*n as i32),\n", - ); - out.push_str( - " crate::proto::data_value::DataType::Price(p) => Some(p.value),\n", - ); - out.push_str( - " crate::proto::data_value::DataType::Timestamp(ts) => Some(crate::decode::timestamp_to_date(ts.epoch_ms)),\n", - ); - out.push_str(" _ => None,\n"); - out.push_str(" })\n"); - out.push_str(" .unwrap_or(0)\n"); - out.push_str(" }\n\n"); - - out.push_str(" // EOD price field: decode to f64 from Price or Number cell.\n"); - out.push_str(" fn eod_price(row: &crate::proto::DataValueList, idx: usize) -> f64 {\n"); - out.push_str(" row.values\n"); - out.push_str(" .get(idx)\n"); - out.push_str(" .and_then(|dv| dv.data_type.as_ref())\n"); - out.push_str(" .map(|dt| match dt {\n"); - out.push_str(" crate::proto::data_value::DataType::Price(p) => {\n"); - out.push_str( - " tdbe::types::price::Price::new(p.value, p.r#type).to_f64()\n", - ); - out.push_str(" }\n"); - out.push_str( - " crate::proto::data_value::DataType::Number(n) => *n as f64,\n", - ); - out.push_str(" _ => 0.0,\n"); - out.push_str(" })\n"); - out.push_str(" .unwrap_or(0.0)\n"); - out.push_str(" }\n\n"); - } - - out.push_str(" let h: Vec<&str> = table.headers.iter().map(|s| s.as_str()).collect();\n"); - - // For eod_style, use the shared find_header() from decode.rs. - if def.eod_style { - out.push_str(" let find = |name: &str| find_header(&h, name);\n\n"); - } - - // Determine which columns need helpers. - let needs_opt_number = def - .columns - .iter() - .any(|c| c.r#type == "i32" && !def.required.contains(&c.name) && c.field != "date"); - let needs_opt_float = def - .columns - .iter() - .any(|c| c.r#type == "f64" && !def.required.contains(&c.name)); - let needs_opt_i64 = def - .columns - .iter() - .any(|c| c.r#type == "i64" && !def.required.contains(&c.name)); - - // Emit opt_number / opt_float / opt_i64 helpers (non-eod only). - if !def.eod_style { - if needs_opt_number { - out.push_str(" fn opt_number(row: &crate::proto::DataValueList, idx: Option) -> i32 {\n"); - out.push_str(" match idx {\n"); - out.push_str(" Some(i) => row_number(row, i),\n"); - out.push_str(" None => 0,\n"); - out.push_str(" }\n"); - out.push_str(" }\n\n"); - } - if needs_opt_float { - out.push_str(" fn opt_float(row: &crate::proto::DataValueList, idx: Option) -> f64 {\n"); - out.push_str(" match idx {\n"); - out.push_str(" Some(i) => row_float(row, i),\n"); - out.push_str(" None => 0.0,\n"); - out.push_str(" }\n"); - out.push_str(" }\n\n"); - } - if needs_opt_i64 { - out.push_str( - " fn opt_i64(row: &crate::proto::DataValueList, idx: Option) -> i64 {\n", - ); - out.push_str(" match idx {\n"); - out.push_str(" Some(i) => row_number_i64(row, i),\n"); - out.push_str(" None => 0,\n"); - out.push_str(" }\n"); - out.push_str(" }\n\n"); - } - } - - // Required header guards (non-eod only). - if !def.eod_style { - for req in &def.required { - let var = idx_var_name(req); - writeln!(out, - " let Some({var}) = find_header(&h, \"{req}\") else {{\n return vec![];\n }};" - ).unwrap(); - } - if !def.required.is_empty() { - out.push('\n'); - } - } - - // Declare index variables for all columns. - for col in &def.columns { - let var = idx_var_name(&col.name); - if def.eod_style { - writeln!(out, " let {var} = find(\"{}\");", col.name).unwrap(); - } else if def.required.contains(&col.name) { - // Already declared above as required. - } else { - writeln!(out, " let {var} = find_header(&h, \"{}\");", col.name).unwrap(); - } - } - - // Contract identification columns (wildcard queries inject expiration/strike/right). - if def.contract_id { - out.push_str(" let _cid_exp_idx = h.iter().position(|c| *c == \"expiration\");\n"); - out.push_str(" let _cid_strike_idx = h.iter().position(|c| *c == \"strike\");\n"); - out.push_str(" let _cid_right_idx = h.iter().position(|c| *c == \"right\");\n"); - out.push_str(" let _cid_strike_is_typed = _cid_strike_idx.is_some() && h.contains(&\"strike\");\n"); - } - - // Check if this is QuoteTick (needs midpoint computation). - let is_quote_tick = type_name == "QuoteTick"; - - out.push('\n'); - - // Emit the row mapping. - out.push_str(" table\n"); - out.push_str(" .data_table\n"); - out.push_str(" .iter()\n"); - out.push_str(" .map(|row| {\n"); - - // Struct literal. - if is_quote_tick { - writeln!(out, " let _tick = {type_name} {{").unwrap(); - } else { - writeln!(out, " {type_name} {{").unwrap(); - } - - for col in &def.columns { - let var = idx_var_name(&col.name); - let is_required = def.required.contains(&col.name); - - match col.r#type.as_str() { - "i32" => { - if def.eod_style { - writeln!( - out, - " {}: {var}.map(|i| eod_num(row, i)).unwrap_or(0),", - col.field - ) - .unwrap(); - } else if is_required { - let accessor = if col.field == "date" { - "row_date" - } else { - "row_number" - }; - writeln!( - out, - " {}: {accessor}(row, {var}),", - col.field - ) - .unwrap(); - } else if col.field == "date" { - writeln!( - out, - " {}: {var}.map(|i| row_date(row, i)).unwrap_or(0),", - col.field - ) - .unwrap(); - } else { - writeln!( - out, - " {}: opt_number(row, {var}),", - col.field - ) - .unwrap(); - } - } - "i64" => { - if is_required { - writeln!( - out, - " {}: row_number_i64(row, {var}),", - col.field - ) - .unwrap(); - } else { - writeln!(out, " {}: opt_i64(row, {var}),", col.field).unwrap(); - } - } - "f64" => { - if is_required { - writeln!(out, " {}: row_float(row, {var}),", col.field).unwrap(); - } else { - writeln!(out, " {}: opt_float(row, {var}),", col.field).unwrap(); - } - } - "String" => { - if is_required { - writeln!(out, " {}: row_text(row, {var}),", col.field).unwrap(); - } else { - writeln!( - out, - " {}: {var}.map(|i| row_text(row, i)).unwrap_or_default(),", - col.field - ) - .unwrap(); - } - } - "price" => { - // Decode price to f64 at parse time. - let field = &col.field; - if is_required { - writeln!(out, " {field}: row_price_f64(row, {var}),").unwrap(); - } else { - writeln!(out, " {field}: match {var} {{").unwrap(); - out.push_str(" Some(i) => row_price_f64(row, i),\n"); - out.push_str(" None => 0.0,\n"); - out.push_str(" },\n"); - } - } - "eod_num" => { - writeln!( - out, - " {}: {var}.map(|i| eod_num(row, i)).unwrap_or(0),", - col.field - ) - .unwrap(); - } - "eod_price" => { - // EOD price field: decode to f64 from Price or Number cell. - writeln!( - out, - " {}: {var}.map(|i| eod_price(row, i)).unwrap_or(0.0),", - col.field - ) - .unwrap(); - } - other => panic!("unknown column type '{other}' in parser for {type_name}"), - } - } - - // Contract identification fields (injected when contract_id = true). - if def.contract_id { - out.push_str( - " expiration: _cid_exp_idx.map(|i| { let n = row_number(row, i); if n != 0 { n } else { let s = row_text(row, i); parse_iso_date(&s) } }).unwrap_or(0),\n", - ); - // strike: decode to f64 - out.push_str(" strike: match _cid_strike_idx {\n"); - out.push_str( - " Some(i) if _cid_strike_is_typed => row_price_f64(row, i),\n", - ); - out.push_str(" Some(i) => row_number(row, i) as f64,\n"); - out.push_str(" None => 0.0,\n"); - out.push_str(" },\n"); - out.push_str(" right: _cid_right_idx.map(|i| {\n"); - out.push_str(" let s = row_text(row, i);\n"); - out.push_str( - " if s.is_empty() { 0i32 } else { s.as_bytes()[0] as i32 }\n", - ); - out.push_str(" }).unwrap_or(0),\n"); - } - - if is_quote_tick { - // QuoteTick gets midpoint computed from bid + ask. - out.push_str(" midpoint: 0.0, // placeholder, set below\n"); - out.push_str(" };\n"); - out.push_str(" let mut _tick = _tick;\n"); - out.push_str(" _tick.midpoint = (_tick.bid + _tick.ask) / 2.0;\n"); - out.push_str(" _tick\n"); - } else { - out.push_str(" }\n"); - } - - out.push_str(" })\n"); - out.push_str(" .collect()\n"); - out.push_str("}\n\n"); -} - -/// Convert a column name to a consistent index variable name. -fn idx_var_name(name: &str) -> String { - format!("{name}_idx") -} - -// ───────────────────────────────────────────────────────────────────────────── -// Code-generate the ENDPOINTS static array from external.proto -// ───────────────────────────────────────────────────────────────────────────── - -/// A parsed proto field. -#[derive(Debug, Clone)] -struct ProtoField { - name: String, - proto_type: String, // "string", "int32", "double", "bool", or "ContractSpec" - is_optional: bool, - is_repeated: bool, -} - -/// A parsed RPC entry. -#[derive(Debug)] -struct Rpc { - rpc_name: String, // e.g. "GetStockHistoryEod" - request_type: String, // e.g. "StockHistoryEodRequest" -} - -#[derive(Debug, Clone)] -struct GeneratedParam { - name: String, - description: String, - param_type: String, - required: bool, -} - -#[derive(Debug, Clone)] -struct GeneratedEndpoint { - name: String, - description: String, - category: String, - subcategory: String, - rest_path: String, - grpc_name: String, - request_type: String, - query_type: String, - fields: Vec, - params: Vec, - return_type: String, -} - -#[derive(Debug, Clone)] -struct ParsedEndpoints { - endpoints: Vec, -} - -/// Parse endpoint metadata from `external.proto` into a reusable intermediate form. -/// -/// This build-time parser performs several tightly-coupled passes over the same -/// proto source: RPC discovery, request-query extraction, field expansion, -/// endpoint normalization, and a small set of SDK-specific augmentations. It is -/// intentionally kept in one place so the generated registry, shared endpoint -/// runtime, and SDK surface stay aligned. -#[allow(clippy::too_many_lines)] // Reason: build-time endpoint parser coordinates multiple passes over one proto source. -fn load_parsed_endpoints() -> Result> { - let proto = std::fs::read_to_string("proto/external.proto")?; - - // ── Parse RPCs ────────────────────────────────────────────────────────── - let rpc_re = regex::Regex::new(r"rpc\s+(Get\w+)\s*\((\w+)\)\s*returns")?; - let rpcs: Vec = rpc_re - .captures_iter(&proto) - .map(|c| Rpc { - rpc_name: c[1].to_string(), - request_type: c[2].to_string(), - }) - .collect(); - - // ── Parse query messages ──────────────────────────────────────────────── - // Everything lives in one package, so ContractSpec is referenced - // unqualified instead of `endpoints.ContractSpec`. - let msg_re = regex::Regex::new(r"message\s+(\w+RequestQuery)\s*\{([^}]*)}")?; - let field_re = regex::Regex::new( - r"(optional\s+|repeated\s+)?(string|int32|double|bool|ContractSpec)\s+(\w+)\s*=\s*\d+", - )?; - - let mut query_messages: HashMap> = HashMap::new(); - for cap in msg_re.captures_iter(&proto) { - let msg_name = cap[1].to_string(); - let body = &cap[2]; - let fields: Vec = field_re - .captures_iter(body) - .map(|f| ProtoField { - name: f[3].to_string(), - proto_type: f[2].to_string(), - is_optional: f.get(1).is_some_and(|m| m.as_str().starts_with("optional")), - is_repeated: f.get(1).is_some_and(|m| m.as_str().starts_with("repeated")), - }) - .collect(); - query_messages.insert(msg_name, fields); - } - - let mut endpoints = Vec::new(); - - for rpc in &rpcs { - // Derive snake_case method name: GetStockHistoryEod → stock_history_eod - let method = rpc_to_method(&rpc.rpc_name); - - // Find the query message: StockHistoryEodRequest → StockHistoryEodRequestQuery - let query_msg_name = format!("{}Query", rpc.request_type); - let fields = if let Some(f) = query_messages.get(&query_msg_name) { - f.clone() - } else { - eprintln!( - "warning: no query message '{}' found, skipping {}", - query_msg_name, rpc.rpc_name - ); - continue; - }; - - // Expand fields (contract_spec → symbol, expiration, strike, right) - let params = expand_fields(&fields); - - let category = derive_category(&method); - let subcategory = derive_subcategory(&method, &category); - let rest_path = derive_rest_path(&method, &category); - let return_type = derive_return_type(&method); - let description = derive_description(&method); - let mut params = params - .into_iter() - .map(|(name, description, param_type, required)| GeneratedParam { - name, - description, - param_type, - required, - }) - .collect::>(); - normalize_method_params(&method, &mut params); - - endpoints.push(GeneratedEndpoint { - name: method, - description, - category, - subcategory, - rest_path, - grpc_name: format!("get_{}", rpc_to_method(&rpc.rpc_name)), - request_type: rpc.request_type.clone(), - query_type: query_msg_name, - fields, - params, - return_type, - }); - } - - // ── Manual extra: stock_history_ohlc_range ───────────────────────────── - // Second SDK-level method on top of the same GetStockHistoryOhlc RPC. - // The proto supports both shapes via the optional `date` vs - // `start_date`/`end_date` fields; the SDK exposes them as two distinct - // methods for nicer ergonomics. The registry parser only picks up one - // method per RPC, so the range variant is appended manually here. - endpoints.push(GeneratedEndpoint { - name: "stock_history_ohlc_range".into(), - description: "Fetch intraday OHLC bars across a date range.".into(), - category: "stock".into(), - subcategory: "history".into(), - rest_path: "/v3/stock/history/ohlc_range".into(), - grpc_name: "get_stock_history_ohlc".into(), - request_type: "StockHistoryOhlcRequest".into(), - query_type: "StockHistoryOhlcRequestQuery".into(), - fields: vec![ - ProtoField { - name: "symbol".into(), - proto_type: "string".into(), - is_optional: false, - is_repeated: false, - }, - ProtoField { - name: "date".into(), - proto_type: "string".into(), - is_optional: true, - is_repeated: false, - }, - ProtoField { - name: "interval".into(), - proto_type: "string".into(), - is_optional: false, - is_repeated: false, - }, - ProtoField { - name: "start_time".into(), - proto_type: "string".into(), - is_optional: true, - is_repeated: false, - }, - ProtoField { - name: "end_time".into(), - proto_type: "string".into(), - is_optional: true, - is_repeated: false, - }, - ProtoField { - name: "venue".into(), - proto_type: "string".into(), - is_optional: true, - is_repeated: false, - }, - ProtoField { - name: "start_date".into(), - proto_type: "string".into(), - is_optional: true, - is_repeated: false, - }, - ProtoField { - name: "end_date".into(), - proto_type: "string".into(), - is_optional: true, - is_repeated: false, - }, - ], - params: vec![ - GeneratedParam { - name: "symbol".into(), - description: "Ticker symbol (e.g. AAPL)".into(), - param_type: "Symbol".into(), - required: true, - }, - GeneratedParam { - name: "start_date".into(), - description: "Start date YYYYMMDD".into(), - param_type: "Date".into(), - required: true, - }, - GeneratedParam { - name: "end_date".into(), - description: "End date YYYYMMDD".into(), - param_type: "Date".into(), - required: true, - }, - GeneratedParam { - name: "interval".into(), - description: "Accepts milliseconds (60000) or shorthand (1m). Presets: 100ms, 500ms, 1s, 5s, 10s, 15s, 30s, 1m, 5m, 10m, 15m, 30m, 1h.".into(), - param_type: "Interval".into(), - required: true, - }, - GeneratedParam { - name: "start_time".into(), - description: "Start time filter (hh:mm:ss.SSS or ms)".into(), - param_type: "Str".into(), - required: false, - }, - GeneratedParam { - name: "end_time".into(), - description: "End time filter (hh:mm:ss.SSS or ms)".into(), - param_type: "Str".into(), - required: false, - }, - GeneratedParam { - name: "venue".into(), - description: "Venue/exchange filter".into(), - param_type: "Str".into(), - required: false, - }, - ], - return_type: "OhlcTicks".into(), - }); - - Ok(ParsedEndpoints { endpoints }) -} - -fn generate_endpoint_registry() -> Result<(), Box> { - let parsed = load_parsed_endpoints()?; - - // ── Generate Rust code ────────────────────────────────────────────────── - let mut code = String::new(); - code.push_str("// Auto-generated by build.rs from external.proto — do not edit manually.\n\n"); - code.push_str("pub static ENDPOINTS: &[EndpointMeta] = &[\n"); - - for endpoint in &parsed.endpoints { - code.push_str(" EndpointMeta {\n"); - writeln!(code, " name: \"{}\",", endpoint.name).unwrap(); - writeln!(code, " description: \"{}\",", endpoint.description).unwrap(); - writeln!(code, " category: \"{}\",", endpoint.category).unwrap(); - writeln!(code, " subcategory: \"{}\",", endpoint.subcategory).unwrap(); - writeln!(code, " rest_path: \"{}\",", endpoint.rest_path).unwrap(); - - if endpoint.params.is_empty() { - code.push_str(" params: &[],\n"); - } else { - code.push_str(" params: &[\n"); - for param in &endpoint.params { - code.push_str(" ParamMeta {\n"); - writeln!(code, " name: \"{}\",", param.name).unwrap(); - writeln!( - code, - " description: \"{}\",", - param.description - ) - .unwrap(); - writeln!( - code, - " param_type: ParamType::{},", - param.param_type - ) - .unwrap(); - writeln!(code, " required: {},", param.required).unwrap(); - code.push_str(" },\n"); - } - code.push_str(" ],\n"); - } - - writeln!( - code, - " returns: ReturnType::{},", - endpoint.return_type - ) - .unwrap(); - code.push_str(" },\n"); - } - - code.push_str("];\n"); - - let out_dir = std::env::var("OUT_DIR")?; - let dest = Path::new(&out_dir).join("registry_generated.rs"); - std::fs::write(&dest, &code)?; - - // Rerun if proto changes - println!("cargo:rerun-if-changed=proto/external.proto"); - - Ok(()) -} - -fn generate_endpoint_runtime() -> Result<(), Box> { - let parsed = load_parsed_endpoints()?; - - let mut code = String::new(); - code.push_str("// Auto-generated by build.rs from external.proto — do not edit manually.\n\n"); - code.push_str("/// Dispatch a validated endpoint call into the generated SDK adapter set.\n"); - code.push_str("///\n"); - code.push_str( - "/// The build script emits one match arm per endpoint from the shared endpoint\n", - ); - code.push_str( - "/// metadata so registry-driven projections stay aligned with the SDK surface.\n", - ); - code.push_str("pub async fn invoke_generated_endpoint(\n"); - code.push_str(" client: &crate::ThetaDataDx,\n"); - code.push_str(" name: &str,\n"); - code.push_str(" args: &EndpointArgs,\n"); - code.push_str(") -> Result {\n"); - code.push_str(" match name {\n"); - - for endpoint in &parsed.endpoints { - generate_endpoint_dispatch_arm(&mut code, endpoint); - } - - code.push_str(" _ => Err(EndpointError::UnknownEndpoint(name.to_string())),\n"); - code.push_str(" }\n"); - code.push_str("}\n"); - - let out_dir = std::env::var("OUT_DIR")?; - let dest = Path::new(&out_dir).join("endpoint_generated.rs"); - std::fs::write(&dest, &code)?; - - println!("cargo:rerun-if-changed=proto/external.proto"); - - Ok(()) -} - -fn generate_direct_endpoints() -> Result<(), Box> { - let parsed = load_parsed_endpoints()?; - - let mut list_code = String::new(); - let mut parsed_code = String::new(); - list_code - .push_str("// Auto-generated by build.rs from external.proto — do not edit manually.\n\n"); - list_code.push_str("impl DirectClient {\n"); - parsed_code - .push_str("// Auto-generated by build.rs from external.proto — do not edit manually.\n\n"); - - for endpoint in &parsed.endpoints { - if is_simple_list_endpoint(&endpoint.name) { - generate_direct_list_endpoint(&mut list_code, endpoint); - } else { - generate_direct_parsed_endpoint(&mut parsed_code, endpoint); - } - } - - let out_dir = std::env::var("OUT_DIR")?; - std::fs::write( - Path::new(&out_dir).join("direct_list_endpoints_generated.rs"), - format!("{list_code}}}\n"), - )?; - std::fs::write( - Path::new(&out_dir).join("direct_parsed_endpoints_generated.rs"), - parsed_code, - )?; - - println!("cargo:rerun-if-changed=proto/external.proto"); - - Ok(()) -} - -fn generate_direct_list_endpoint(out: &mut String, endpoint: &GeneratedEndpoint) { - writeln!(out, "list_endpoint! {{").unwrap(); - writeln!(out, " #[doc = {:?}]", endpoint.description).unwrap(); - writeln!( - out, - " #[doc = {:?}]", - format!("gRPC stub: `{}`", endpoint.grpc_name) - ) - .unwrap(); - - let signature = endpoint - .params - .iter() - .map(|param| format!("{}: &str", direct_method_arg_name(endpoint, param))) - .collect::>() - .join(", "); - if signature.is_empty() { - writeln!( - out, - " fn {}() -> {:?};", - endpoint.name, - direct_list_column(endpoint) - ) - .unwrap(); - } else { - writeln!( - out, - " fn {}({signature}) -> {:?};", - endpoint.name, - direct_list_column(endpoint) - ) - .unwrap(); - } - - writeln!(out, " grpc: {};", endpoint.grpc_name).unwrap(); - writeln!(out, " request: {};", endpoint.request_type).unwrap(); - if endpoint.fields.is_empty() { - writeln!(out, " query: {} {{}};", endpoint.query_type).unwrap(); - } else { - writeln!(out, " query: {} {{", endpoint.query_type).unwrap(); - for field in &endpoint.fields { - writeln!( - out, - " {}: {},", - field.name, - direct_query_field_expr(endpoint, field, true) - ) - .unwrap(); - } - out.push_str(" };\n"); - } - out.push_str("}\n\n"); -} - -fn generate_direct_parsed_endpoint(out: &mut String, endpoint: &GeneratedEndpoint) { - writeln!(out, "parsed_endpoint! {{").unwrap(); - writeln!(out, " #[doc = {:?}]", endpoint.description).unwrap(); - writeln!( - out, - " #[doc = {:?}]", - format!("gRPC stub: `{}`", endpoint.grpc_name) - ) - .unwrap(); - writeln!( - out, - " builder {}Builder;", - to_pascal_case(&endpoint.name) - ) - .unwrap(); - - let method_params = endpoint - .params - .iter() - .filter(|param| is_method_call_param(param)) - .collect::>(); - let signature = method_params - .iter() - .map(|param| { - format!( - "{}: {}", - direct_method_arg_name(endpoint, param), - direct_required_kind(param) - ) - }) - .collect::>() - .join(", "); - writeln!( - out, - " fn {}({signature}) -> {};", - endpoint.name, - direct_return_type(&endpoint.return_type) - ) - .unwrap(); - - writeln!(out, " grpc: {};", endpoint.grpc_name).unwrap(); - writeln!(out, " request: {};", endpoint.request_type).unwrap(); - if endpoint.fields.is_empty() { - writeln!(out, " query: {} {{}};", endpoint.query_type).unwrap(); - } else { - writeln!(out, " query: {} {{", endpoint.query_type).unwrap(); - for field in &endpoint.fields { - writeln!( - out, - " {}: {},", - field.name, - direct_query_field_expr(endpoint, field, false) - ) - .unwrap(); - } - out.push_str(" };\n"); - } - writeln!( - out, - " parse: {};", - direct_parser_name(&endpoint.return_type) - ) - .unwrap(); - - let date_args = method_params - .iter() - .filter_map(|param| direct_date_arg_name(endpoint, param)) - .collect::>(); - if !date_args.is_empty() { - writeln!(out, " dates: {};", date_args.join(", ")).unwrap(); - } - - let optional_params = endpoint - .params - .iter() - .filter(|param| !is_method_call_param(param)) - .collect::>(); - if optional_params.is_empty() { - out.push_str(" optional {}\n"); - } else { - out.push_str(" optional {\n"); - for param in optional_params { - let (kind, default) = direct_optional_kind_and_default(param); - writeln!(out, " {}: {} = {},", param.name, kind, default).unwrap(); - } - out.push_str(" }\n"); - } - out.push_str("}\n\n"); -} - -fn direct_method_arg_name(endpoint: &GeneratedEndpoint, param: &GeneratedParam) -> String { - if param.param_type == "Symbols" { - return "symbols".into(); - } - - match (endpoint.name.as_str(), param.name.as_str()) { - ("stock_history_eod" | "option_history_eod" | "index_history_eod", "start_date") => { - "start".into() - } - ("stock_history_eod" | "option_history_eod" | "index_history_eod", "end_date") => { - "end".into() - } - _ => param.name.clone(), - } -} - -fn direct_date_arg_name(endpoint: &GeneratedEndpoint, param: &GeneratedParam) -> Option { - match param.name.as_str() { - "date" | "start_date" | "end_date" => Some(direct_method_arg_name(endpoint, param)), - _ => None, - } -} - -fn direct_required_kind(param: &GeneratedParam) -> &'static str { - if param.param_type == "Symbols" { - "str_vec" - } else { - "str" - } -} - -fn direct_optional_kind_and_default(param: &GeneratedParam) -> (&'static str, &'static str) { - match param.name.as_str() { - "start_time" => ("string", "\"09:30:00\".to_string()"), - "end_time" => ("string", "\"16:00:00\".to_string()"), - _ => match param.param_type.as_str() { - "Int" => ("opt_i32", "None"), - "Float" => ("opt_f64", "None"), - "Bool" => ("opt_bool", "None"), - _ => ("opt_str", "None"), - }, - } -} - -fn direct_list_column(endpoint: &GeneratedEndpoint) -> &'static str { - if endpoint.name.ends_with("_list_symbols") { - "symbol" - } else if endpoint.name.ends_with("_list_dates") { - "date" - } else if endpoint.name.ends_with("_list_expirations") { - "expiration" - } else if endpoint.name.ends_with("_list_strikes") { - "strike" - } else { - "value" - } -} - -fn direct_return_type(return_type: &str) -> &'static str { - match return_type { - "EodTicks" => "Vec", - "OhlcTicks" => "Vec", - "TradeTicks" => "Vec", - "QuoteTicks" => "Vec", - "TradeQuoteTicks" => "Vec", - "OpenInterestTicks" => "Vec", - "MarketValueTicks" => "Vec", - "GreeksTicks" => "Vec", - "IvTicks" => "Vec", - "PriceTicks" => "Vec", - "CalendarDays" => "Vec", - "InterestRateTicks" => "Vec", - "OptionContracts" => "Vec", - other => panic!("unsupported direct return type: {other}"), - } -} - -fn direct_parser_name(return_type: &str) -> &'static str { - match return_type { - "EodTicks" => "decode::parse_eod_ticks", - "OhlcTicks" => "decode::parse_ohlc_ticks", - "TradeTicks" => "decode::parse_trade_ticks", - "QuoteTicks" => "decode::parse_quote_ticks", - "TradeQuoteTicks" => "decode::parse_trade_quote_ticks", - "OpenInterestTicks" => "decode::parse_open_interest_ticks", - "MarketValueTicks" => "decode::parse_market_value_ticks", - "GreeksTicks" => "decode::parse_greeks_ticks", - "IvTicks" => "decode::parse_iv_ticks", - "PriceTicks" => "decode::parse_price_ticks", - "CalendarDays" => "decode::parse_calendar_days_v3", - "InterestRateTicks" => "decode::parse_interest_rate_ticks", - "OptionContracts" => "decode::parse_option_contracts_v3", - other => panic!("unsupported parser return type: {other}"), - } -} - -fn direct_query_field_expr( - endpoint: &GeneratedEndpoint, - field: &ProtoField, - list_context: bool, -) -> String { - if field.proto_type == "ContractSpec" { - return "contract_spec!(symbol, expiration, strike, right)".into(); - } - if field.name == "date" && endpoint.name == "stock_history_ohlc_range" { - return "None".into(); - } - - let param = endpoint - .params - .iter() - .find(|param| param.name == field.name) - .unwrap_or_else(|| { - panic!( - "missing param metadata for {}::{}", - endpoint.name, field.name - ) - }); - let arg_name = direct_method_arg_name(endpoint, param); - let is_method_param = is_method_call_param(param); - - match field.name.as_str() { - "symbol" if field.is_repeated => { - if param.param_type == "Symbols" { - "symbols.clone()".into() - } else if list_context { - format!("vec![{arg_name}.to_string()]") - } else { - format!("vec![{arg_name}.clone()]") - } - } - "interval" => format!("normalize_interval(&{arg_name})"), - "start_time" | "end_time" => format!("Some({arg_name}.clone())"), - "venue" if endpoint.category == "stock" => { - "venue.clone().or_else(|| Some(\"nqb\".to_string()))".into() - } - _ if field.proto_type == "string" => { - if field.is_optional { - if is_method_param { - format!("Some({arg_name}.clone())") - } else { - format!("{arg_name}.clone()") - } - } else if list_context { - format!("{arg_name}.to_string()") - } else { - format!("{arg_name}.clone()") - } - } - _ => arg_name, - } -} - -fn to_pascal_case(value: &str) -> String { - value - .split('_') - .filter(|segment| !segment.is_empty()) - .map(|segment| { - let mut chars = segment.chars(); - match chars.next() { - Some(first) => first.to_uppercase().to_string() + chars.as_str(), - None => String::new(), - } - }) - .collect::() -} - -fn generate_endpoint_dispatch_arm(out: &mut String, endpoint: &GeneratedEndpoint) { - writeln!(out, " \"{}\" => {{", endpoint.name).unwrap(); - - if is_simple_list_endpoint(&endpoint.name) { - for param in &endpoint.params { - emit_required_arg(out, endpoint, param); - } - let args = endpoint - .params - .iter() - .map(call_arg_name) - .collect::>() - .join(", "); - writeln!( - out, - " let values = client.{}({args}).await?;", - endpoint.name - ) - .unwrap(); - writeln!( - out, - " Ok(EndpointOutput::{}(values))", - endpoint.return_type - ) - .unwrap(); - out.push_str(" }\n"); - return; - } - - let method_call_params = endpoint - .params - .iter() - .filter(|param| is_method_call_param(param)) - .collect::>(); - let builder_params = endpoint - .params - .iter() - .filter(|param| !is_method_call_param(param)) - .collect::>(); - - for param in &method_call_params { - emit_required_arg(out, endpoint, param); - } - - let call_args = method_call_params - .into_iter() - .map(call_arg_name) - .collect::>() - .join(", "); - - if builder_params.is_empty() { - writeln!( - out, - " let result = client.{}({call_args}).await?;", - endpoint.name - ) - .unwrap(); - } else { - writeln!( - out, - " let mut builder = client.{}({call_args});", - endpoint.name - ) - .unwrap(); - - for param in builder_params { - let getter = optional_getter_name(¶m.param_type); - writeln!( - out, - " if let Some(value) = args.{getter}(\"{}\")? {{", - param.name - ) - .unwrap(); - writeln!( - out, - " builder = builder.{}(value);", - param.name - ) - .unwrap(); - out.push_str(" }\n"); - } - - out.push_str(" let result = builder.await?;\n"); - } - writeln!( - out, - " Ok(EndpointOutput::{}(result))", - endpoint.return_type - ) - .unwrap(); - out.push_str(" }\n"); -} - -fn emit_required_arg(out: &mut String, endpoint: &GeneratedEndpoint, param: &GeneratedParam) { - if endpoint.name == "stock_history_ohlc" && param.name == "date" && !param.required { - out.push_str(" let date = args.optional_date(\"date\")?.unwrap_or(\"\");\n"); - return; - } - - if param.param_type == "Symbols" { - writeln!( - out, - " let symbol_values = args.required_symbols(\"{}\")?;", - param.name - ) - .unwrap(); - out.push_str( - " let symbol_refs: Vec<&str> = symbol_values.iter().map(String::as_str).collect();\n", - ); - return; - } - - let getter = required_getter_name(¶m.param_type); - writeln!( - out, - " let {} = args.{getter}(\"{}\")?;", - param.name, param.name - ) - .unwrap(); -} - -fn call_arg_name(param: &GeneratedParam) -> String { - if param.param_type == "Symbols" { - "&symbol_refs".into() - } else { - param.name.clone() - } -} - -fn is_method_call_param(param: &GeneratedParam) -> bool { - // The proto models many intraday history shapes as a union of - // `date`-based and `start_date`/`end_date`-based queries, so `date` - // often appears optional at the wire level even when the SDK method - // exposes it as a positional constructor argument. - param.required || param.name == "date" -} - -fn normalize_method_params(method: &str, params: &mut [GeneratedParam]) { - let supports_symbol_lists = - method.starts_with("stock_snapshot_") || method.starts_with("index_snapshot_"); - - if !supports_symbol_lists { - for param in params.iter_mut() { - if param.name == "symbol" && param.param_type == "Symbols" { - param.param_type = "Symbol".into(); - param.description = "Ticker symbol (e.g. AAPL)".into(); - } - } - } -} - -fn required_getter_name(param_type: &str) -> &'static str { - match param_type { - "Symbol" => "required_symbol", - "Symbols" => "required_symbols", - "Date" | "Expiration" => "required_date", - "Interval" => "required_interval", - "Right" => "required_right", - "Int" => "required_int32", - "Float" => "required_float64", - "Bool" => "required_bool", - "Year" => "required_year", - _ => "required_str", - } -} - -fn optional_getter_name(param_type: &str) -> &'static str { - match param_type { - "Date" | "Expiration" => "optional_date", - "Int" => "optional_int32", - "Float" => "optional_float64", - "Bool" => "optional_bool", - _ => "optional_str", - } -} - -fn is_simple_list_endpoint(method: &str) -> bool { - method.ends_with("_list_symbols") - || method.ends_with("_list_dates") - || method.ends_with("_list_expirations") - || method.ends_with("_list_strikes") -} - -/// Convert `GetStockHistoryEod` → `stock_history_eod`. -fn rpc_to_method(rpc_name: &str) -> String { - // Strip leading "Get" - let name = rpc_name.strip_prefix("Get").unwrap_or(rpc_name); // build script: panic is intentional - // PascalCase → snake_case - let mut result = String::new(); - for (i, ch) in name.chars().enumerate() { - if ch.is_uppercase() { - if i > 0 { - result.push('_'); - } - result.push(ch.to_lowercase().next().unwrap()); // build script: panic is intentional - } else { - result.push(ch); - } - } - result -} - -/// Expand proto fields, replacing `contract_spec` with (symbol, expiration, strike, right). -/// -/// Many option query messages carry both a `ContractSpec` (contract identity, -/// expanded here to 4 fields) AND an explicit top-level `expiration` field -/// (the query range expiration — e.g. "include all contracts expiring by..."), -/// which would otherwise collide with the contract's own expiration. Any -/// post-expansion duplicate parameter name is dropped in favor of the first -/// occurrence (ContractSpec wins, since it is structurally the contract -/// identity the user really cares about). -fn expand_fields(fields: &[ProtoField]) -> Vec<(String, String, String, bool)> { - let mut params: Vec<(String, String, String, bool)> = Vec::new(); - let mut seen: std::collections::HashSet = std::collections::HashSet::new(); - - let push = |params: &mut Vec<(String, String, String, bool)>, - seen: &mut std::collections::HashSet, - entry: (String, String, String, bool)| { - if seen.insert(entry.0.clone()) { - params.push(entry); - } - }; - - for f in fields { - if f.proto_type == "ContractSpec" { - // Expand to the 4 contract spec fields (symbol, expiration, strike, right). - push( - &mut params, - &mut seen, - ( - "symbol".into(), - "Underlying symbol (e.g. AAPL)".into(), - "Symbol".into(), - true, - ), - ); - push( - &mut params, - &mut seen, - ( - "expiration".into(), - "Expiration date YYYYMMDD".into(), - "Expiration".into(), - true, - ), - ); - push( - &mut params, - &mut seen, - ( - "strike".into(), - "Strike price (raw integer)".into(), - "Strike".into(), - true, - ), - ); - push( - &mut params, - &mut seen, - ( - "right".into(), - "C for call, P for put".into(), - "Right".into(), - true, - ), - ); - } else { - let (param_type, desc) = map_field(&f.name, &f.proto_type, f.is_repeated); - let required = !f.is_optional; - push( - &mut params, - &mut seen, - (f.name.clone(), desc, param_type, required), - ); - } - } - params -} - -/// Map a proto field (name + type + repeated) to (`ParamType` variant name, description). -fn map_field(name: &str, proto_type: &str, is_repeated: bool) -> (String, String) { - // Repeated string symbol → Symbols - if is_repeated && name == "symbol" { - return ( - "Symbols".into(), - "Comma-separated ticker symbols (e.g. AAPL,MSFT)".into(), - ); - } - - match (proto_type, name) { - ("string", "symbol") => ("Symbol".into(), "Ticker symbol (e.g. AAPL)".into()), - ("string", "start_date") => ("Date".into(), "Start date YYYYMMDD".into()), - ("string", "end_date") => ("Date".into(), "End date YYYYMMDD".into()), - ("string", "date") => ("Date".into(), "Date YYYYMMDD".into()), - ("string", "interval") => ( - "Interval".into(), - "Accepts milliseconds (60000) or shorthand (1m). Presets: 100ms, 500ms, 1s, 5s, 10s, 15s, 30s, 1m, 5m, 10m, 15m, 30m, 1h.".into(), - ), - ("string", "right") => ("Right".into(), "C for call, P for put".into()), - ("string", "strike") => ("Strike".into(), "Strike price (raw integer)".into()), - ("string", "expiration") => ("Expiration".into(), "Expiration date YYYYMMDD".into()), - ("string", "request_type") => ( - "RequestType".into(), - "Request type: EOD, TRADE, QUOTE, OHLC, etc.".into(), - ), - ("string", "year") => ("Year".into(), "4-digit year (e.g. 2024)".into()), - ("string", "time_of_day") => ( - "Str".into(), - "Milliseconds from midnight ET (e.g. 34200000 = 9:30 AM)".into(), - ), - ("string", "venue") => ("Str".into(), "Venue/exchange filter".into()), - ("string", "min_time") => ("Str".into(), "Minimum time filter".into()), - ("string", "start_time") => ("Str".into(), "Start time filter".into()), - ("string", "end_time") => ("Str".into(), "End time filter".into()), - ("string", "rate_type") => ("Str".into(), "Rate type".into()), - ("string", "version") => ("Str".into(), "Greeks model version".into()), - ("double", _) => ("Float".into(), humanize_name(name).clone()), - ("int32", "max_dte") => ("Int".into(), "Maximum days to expiration".into()), - ("int32", "strike_range") => ("Int".into(), "Strike range filter".into()), - ("int32", _) => ("Int".into(), humanize_name(name).clone()), - ("bool", "exclusive") => ("Bool".into(), "Exclusive time boundary".into()), - ("bool", "use_market_value") => ("Bool".into(), "Use market value for Greeks".into()), - ("bool", "underlyer_use_nbbo") => ("Bool".into(), "Use NBBO for underlyer price".into()), - ("bool", _) => ("Bool".into(), humanize_name(name).clone()), - _ => ("Str".into(), humanize_name(name).clone()), - } -} - -fn humanize_name(name: &str) -> String { - name.replace('_', " ") - .split_whitespace() - .enumerate() - .map(|(i, w)| { - if i == 0 { - let mut c = w.chars(); - match c.next() { - Some(first) => first.to_uppercase().to_string() + c.as_str(), - None => String::new(), - } - } else { - w.to_string() - } - }) - .collect::>() - .join(" ") -} - -fn derive_category(method: &str) -> String { - if method.starts_with("stock_") { - "stock".into() - } else if method.starts_with("option_") { - "option".into() - } else if method.starts_with("index_") { - "index".into() - } else if method.starts_with("calendar_") { - "calendar".into() - } else if method.starts_with("interest_rate_") { - "rate".into() - } else { - "other".into() - } -} - -fn derive_subcategory(method: &str, category: &str) -> String { - // Strip the category prefix to get the rest - let rest = match category { - // build script: panic is intentional (unwrap_or provides safe fallback) - "stock" => method.strip_prefix("stock_").unwrap_or(method), - "option" => method.strip_prefix("option_").unwrap_or(method), - "index" => method.strip_prefix("index_").unwrap_or(method), - "calendar" => method.strip_prefix("calendar_").unwrap_or(method), - "rate" => method.strip_prefix("interest_rate_").unwrap_or(method), - _ => method, - }; - - if rest.starts_with("list_") { - "list".into() - } else if rest.starts_with("snapshot_greeks_") || rest.starts_with("snapshot_greeks") { - "snapshot_greeks".into() - } else if rest.starts_with("snapshot_") { - "snapshot".into() - } else if rest.starts_with("history_trade_greeks_") { - "history_trade_greeks".into() - } else if rest.starts_with("history_greeks_") { - "history_greeks".into() - } else if rest.starts_with("history_") { - "history".into() - } else if rest.starts_with("at_time_") { - "at_time".into() - } else if rest == "open_today" { - "status".into() - } else if rest == "on_date" || rest == "year" { - "query".into() - } else { - "other".into() - } -} - -fn derive_rest_path(method: &str, category: &str) -> String { - let rest = match category { - "rate" => method.strip_prefix("interest_rate_").unwrap_or(method), - _ => method - .strip_prefix(&format!("{category}_")) - .unwrap_or(method), - }; - - let path_rest = if let Some(what) = rest.strip_prefix("history_trade_greeks_") { - format!("history/trade_greeks/{what}") - } else if let Some(what) = rest.strip_prefix("history_greeks_") { - format!("history/greeks/{what}") - } else if let Some(what) = rest.strip_prefix("snapshot_greeks_") { - format!("snapshot/greeks/{what}") - } else if let Some(what) = rest.strip_prefix("history_") { - format!("history/{what}") - } else if let Some(what) = rest.strip_prefix("snapshot_") { - format!("snapshot/{what}") - } else if let Some(what) = rest.strip_prefix("list_") { - format!("list/{what}") - } else if let Some(what) = rest.strip_prefix("at_time_") { - format!("at_time/{what}") - } else { - rest.to_string() - }; - - format!("/v3/{category}/{path_rest}") -} - -fn derive_return_type(method: &str) -> String { - if is_simple_list_endpoint(method) { - return "StringList".into(); - } - - if method == "option_list_contracts" { - return "OptionContracts".into(); - } - - if method.starts_with("calendar_") { - return "CalendarDays".into(); - } - - if method.starts_with("interest_rate_") { - return "InterestRateTicks".into(); - } - - if method.contains("_trade_quote") { - return "TradeQuoteTicks".into(); - } - - if method.contains("_open_interest") { - return "OpenInterestTicks".into(); - } - - if method.contains("_market_value") { - return "MarketValueTicks".into(); - } - - if method.contains("greeks_implied_volatility") { - return "IvTicks".into(); - } - - if method.contains("_greeks_") { - return "GreeksTicks".into(); - } - - if method == "index_snapshot_price" - || method == "index_history_price" - || method == "index_at_time_price" - { - return "PriceTicks".into(); - } - - if method.ends_with("_history_eod") { - return "EodTicks".into(); - } - - if method.contains("_ohlc") { - return "OhlcTicks".into(); - } - - if method.contains("_trade") || method.ends_with("_trade") { - return "TradeTicks".into(); - } - - if method.contains("_quote") || method.ends_with("_quote") { - return "QuoteTicks".into(); - } - - panic!("unhandled return type mapping for endpoint {method}"); -} - -// Reason: one match dispatch per endpoint — cannot be meaningfully split. -#[allow(clippy::too_many_lines)] -fn derive_description(method: &str) -> String { - // Hand-crafted descriptions for known patterns - match method { - // Stock List - "stock_list_symbols" => "List all available stock ticker symbols.".into(), - "stock_list_dates" => { - "List available dates for a stock by request type (EOD, TRADE, QUOTE, etc.).".into() - } - // Stock Snapshot - "stock_snapshot_ohlc" => "Get the latest OHLC snapshot for one or more stocks.".into(), - "stock_snapshot_trade" => "Get the latest trade snapshot for one or more stocks.".into(), - "stock_snapshot_quote" => { - "Get the latest NBBO quote snapshot for one or more stocks.".into() - } - "stock_snapshot_market_value" => { - "Get the latest market value snapshot for one or more stocks.".into() - } - // Stock History - "stock_history_eod" => { - "Fetch end-of-day stock data for a date range. Returns OHLCV + bid/ask per trading day." - .into() - } - "stock_history_ohlc" => "Fetch intraday OHLC bars for a stock on a single date.".into(), - "stock_history_trade" => "Fetch all trades for a stock on a given date.".into(), - "stock_history_quote" => { - "Fetch NBBO quotes for a stock on a given date at a given interval.".into() - } - "stock_history_trade_quote" => { - "Fetch combined trade + quote ticks for a stock on a given date. Returns raw DataTable." - .into() - } - // Stock At-Time - "stock_at_time_trade" => { - "Fetch the trade at a specific time of day across a date range.".into() - } - "stock_at_time_quote" => { - "Fetch the quote at a specific time of day across a date range.".into() - } - // Option List - "option_list_symbols" => "List all available option underlying symbols.".into(), - "option_list_dates" => { - "List available dates for an option contract by request type.".into() - } - "option_list_expirations" => { - "List available expiration dates for an option underlying.".into() - } - "option_list_strikes" => { - "List available strike prices for an option at a given expiration.".into() - } - "option_list_contracts" => "List all option contracts for a symbol on a given date.".into(), - // Option Snapshot - "option_snapshot_ohlc" => "Get the latest OHLC snapshot for an option contract.".into(), - "option_snapshot_trade" => "Get the latest trade snapshot for an option contract.".into(), - "option_snapshot_quote" => { - "Get the latest NBBO quote snapshot for an option contract.".into() - } - "option_snapshot_open_interest" => { - "Get the latest open interest snapshot for an option contract.".into() - } - "option_snapshot_market_value" => { - "Get the latest market value snapshot for an option contract.".into() - } - // Option Snapshot Greeks - "option_snapshot_greeks_implied_volatility" => { - "Get implied volatility snapshot for an option contract (from ThetaData server).".into() - } - "option_snapshot_greeks_all" => { - "Get all Greeks snapshot for an option contract (from ThetaData server).".into() - } - "option_snapshot_greeks_first_order" => { - "Get first-order Greeks snapshot (delta, theta, rho) for an option contract.".into() - } - "option_snapshot_greeks_second_order" => { - "Get second-order Greeks snapshot (gamma, vanna, charm) for an option contract.".into() - } - "option_snapshot_greeks_third_order" => { - "Get third-order Greeks snapshot (speed, color, ultima) for an option contract.".into() - } - // Option History - "option_history_eod" => { - "Fetch end-of-day option data for a contract over a date range.".into() - } - "option_history_ohlc" => "Fetch intraday OHLC bars for an option contract.".into(), - "option_history_trade" => "Fetch all trades for an option contract on a given date.".into(), - "option_history_quote" => { - "Fetch NBBO quotes for an option contract on a given date.".into() - } - "option_history_trade_quote" => { - "Fetch combined trade + quote ticks for an option contract.".into() - } - "option_history_open_interest" => { - "Fetch open interest history for an option contract.".into() - } - // Option History Greeks - "option_history_greeks_eod" => { - "Fetch end-of-day Greeks history for an option contract.".into() - } - "option_history_greeks_all" => { - "Fetch all Greeks history for an option contract (intraday, sampled by interval)." - .into() - } - "option_history_trade_greeks_all" => { - "Fetch all Greeks on each trade for an option contract.".into() - } - "option_history_greeks_first_order" => { - "Fetch first-order Greeks history (intraday, sampled by interval).".into() - } - "option_history_trade_greeks_first_order" => { - "Fetch first-order Greeks on each trade for an option contract.".into() - } - "option_history_greeks_second_order" => { - "Fetch second-order Greeks history (intraday, sampled by interval).".into() - } - "option_history_trade_greeks_second_order" => { - "Fetch second-order Greeks on each trade for an option contract.".into() - } - "option_history_greeks_third_order" => { - "Fetch third-order Greeks history (intraday, sampled by interval).".into() - } - "option_history_trade_greeks_third_order" => { - "Fetch third-order Greeks on each trade for an option contract.".into() - } - "option_history_greeks_implied_volatility" => { - "Fetch implied volatility history (intraday, sampled by interval).".into() - } - "option_history_trade_greeks_implied_volatility" => { - "Fetch implied volatility on each trade for an option contract.".into() - } - // Option At-Time - "option_at_time_trade" => { - "Fetch the trade at a specific time of day across a date range for an option.".into() - } - "option_at_time_quote" => { - "Fetch the quote at a specific time of day across a date range for an option.".into() - } - // Index - "index_list_symbols" => "List all available index symbols.".into(), - "index_list_dates" => "List available dates for an index symbol.".into(), - "index_snapshot_ohlc" => "Get the latest OHLC snapshot for one or more indices.".into(), - "index_snapshot_price" => "Get the latest price snapshot for one or more indices.".into(), - "index_snapshot_market_value" => { - "Get the latest market value snapshot for one or more indices.".into() - } - "index_history_eod" => "Fetch end-of-day index data for a date range.".into(), - "index_history_ohlc" => "Fetch intraday OHLC bars for an index.".into(), - "index_history_price" => "Fetch intraday price history for an index.".into(), - "index_at_time_price" => { - "Fetch the index price at a specific time of day across a date range.".into() - } - // Calendar - "calendar_open_today" => "Check whether the market is open today.".into(), - "calendar_on_date" => "Get calendar information for a specific date.".into(), - "calendar_year" => "Get calendar information for an entire year.".into(), - // Rate - "interest_rate_history_eod" => "Fetch end-of-day interest rate history.".into(), - // Fallback: auto-generate from method name - _ => { - let words = method.replace('_', " "); - let mut chars = words.chars(); - match chars.next() { - Some(c) => { - format!("{}{}.", c.to_uppercase(), chars.as_str()) - } - None => method.to_string(), - } - } - } + build_support::run() } diff --git a/crates/thetadatadx/build_support/endpoints.rs b/crates/thetadatadx/build_support/endpoints.rs new file mode 100644 index 000000000..b743df653 --- /dev/null +++ b/crates/thetadatadx/build_support/endpoints.rs @@ -0,0 +1,2053 @@ +//! Endpoint surface generation and validation. +//! +//! This module treats `endpoint_surface.toml` as the checked-in source of truth +//! for the normalized SDK surface, while still validating each declared +//! endpoint against the upstream gRPC wire contract in `proto/external.proto`. +//! The resulting joined model drives generated registry metadata, the shared +//! endpoint runtime, and non-streaming `DirectClient` methods. + +use std::collections::{HashMap, HashSet}; +use std::fmt::Write as _; +use std::path::Path; + +use serde::Deserialize; + +/// A checked-in endpoint surface specification file. +#[derive(Debug, Deserialize)] +#[serde(deny_unknown_fields)] +struct SurfaceSpec { + version: u32, + #[serde(default)] + param_groups: HashMap, + #[serde(default)] + templates: HashMap, + endpoints: Vec, +} + +/// A reusable parameter group declared in `endpoint_surface.toml`. +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +struct SurfaceParamGroup { + #[serde(default)] + params: Vec, +} + +/// A reusable endpoint template declared in `endpoint_surface.toml`. +#[derive(Debug, Clone, Default, Deserialize)] +#[serde(deny_unknown_fields)] +struct SurfaceTemplate { + #[serde(default)] + extends: Option, + #[serde(default)] + wire_name: Option, + #[serde(default)] + description: Option, + #[serde(default)] + category: Option, + #[serde(default)] + subcategory: Option, + #[serde(default)] + rest_path: Option, + #[serde(default)] + kind: Option, + #[serde(default)] + returns: Option, + #[serde(default)] + list_column: Option, + #[serde(default)] + params: Vec, +} + +/// A normalized endpoint surface entry loaded from `endpoint_surface.toml`. +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +struct SurfaceEndpoint { + name: String, + #[serde(default)] + template: Option, + #[serde(default)] + wire_name: Option, + #[serde(default)] + description: Option, + #[serde(default)] + category: Option, + #[serde(default)] + subcategory: Option, + #[serde(default)] + rest_path: Option, + #[serde(default)] + kind: Option, + #[serde(default)] + returns: Option, + #[serde(default)] + list_column: Option, + #[serde(default)] + params: Vec, +} + +/// A normalized endpoint parameter entry loaded from `endpoint_surface.toml`. +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +struct SurfaceParam { + name: String, + description: String, + param_type: String, + required: bool, + binding: String, + #[serde(default)] + arg_name: Option, + #[serde(default)] + default: Option, +} + +/// A single parameter entry or reference inside a parameter group, template, or endpoint. +#[derive(Debug, Clone, Deserialize)] +#[serde(untagged)] +enum SurfaceParamEntry { + Use(SurfaceParamUse), + Param(SurfaceParam), +} + +/// A reference to a reusable parameter group. +#[derive(Debug, Clone, Deserialize)] +#[serde(deny_unknown_fields)] +struct SurfaceParamUse { + #[serde(rename = "use")] + group: String, +} + +#[derive(Debug, Clone, Default)] +struct ResolvedTemplate { + wire_name: Option, + description: Option, + category: Option, + subcategory: Option, + rest_path: Option, + kind: Option, + returns: Option, + list_column: Option, + params: Vec, +} + +#[derive(Debug, Clone)] +struct ResolvedSurfaceEndpoint { + name: String, + wire_name: Option, + description: String, + category: String, + subcategory: String, + rest_path: String, + kind: String, + returns: String, + list_column: Option, + params: Vec, +} + +/// A parsed proto field. +#[derive(Debug, Clone)] +struct ProtoField { + name: String, + proto_type: String, // "string", "int32", "double", "bool", or "ContractSpec" + is_optional: bool, + is_repeated: bool, +} + +/// A parsed RPC entry. +#[derive(Debug)] +struct Rpc { + rpc_name: String, // e.g. "GetStockHistoryEod" + request_type: String, // e.g. "StockHistoryEodRequest" +} + +#[derive(Debug, Clone)] +struct GeneratedParam { + name: String, + description: String, + param_type: String, + required: bool, + binding: String, + arg_name: Option, + default: Option, +} + +#[derive(Debug, Clone)] +struct GeneratedEndpoint { + name: String, + description: String, + category: String, + subcategory: String, + rest_path: String, + grpc_name: String, + request_type: String, + query_type: String, + fields: Vec, + params: Vec, + return_type: String, + kind: String, + list_column: Option, +} + +#[derive(Debug, Clone)] +struct ParsedEndpoints { + endpoints: Vec, +} + +/// Parse endpoint metadata from `external.proto` into a reusable intermediate form. +/// +/// This build-time parser performs several tightly-coupled passes over the same +/// proto source: RPC discovery, request-query extraction, field expansion, +/// endpoint normalization, and a small set of SDK-specific augmentations. It is +/// intentionally kept in one place so the generated registry, shared endpoint +/// runtime, and SDK surface stay aligned while the explicit endpoint surface +/// spec is validated against the wire contract. +#[allow(clippy::too_many_lines)] // Reason: build-time endpoint parser coordinates multiple passes over one proto source. +fn load_proto_endpoints() -> Result> { + let proto = std::fs::read_to_string("proto/external.proto")?; + + // ── Parse RPCs ────────────────────────────────────────────────────────── + let rpc_re = regex::Regex::new(r"rpc\s+(Get\w+)\s*\((\w+)\)\s*returns")?; + let rpcs: Vec = rpc_re + .captures_iter(&proto) + .map(|c| Rpc { + rpc_name: c[1].to_string(), + request_type: c[2].to_string(), + }) + .collect(); + + // ── Parse query messages ──────────────────────────────────────────────── + // Everything lives in one package, so ContractSpec is referenced + // unqualified instead of `endpoints.ContractSpec`. + let msg_re = regex::Regex::new(r"message\s+(\w+RequestQuery)\s*\{([^}]*)}")?; + let field_re = regex::Regex::new( + r"(optional\s+|repeated\s+)?(string|int32|double|bool|ContractSpec)\s+(\w+)\s*=\s*\d+", + )?; + + let mut query_messages: HashMap> = HashMap::new(); + for cap in msg_re.captures_iter(&proto) { + let msg_name = cap[1].to_string(); + let body = &cap[2]; + let fields: Vec = field_re + .captures_iter(body) + .map(|f| ProtoField { + name: f[3].to_string(), + proto_type: f[2].to_string(), + is_optional: f.get(1).is_some_and(|m| m.as_str().starts_with("optional")), + is_repeated: f.get(1).is_some_and(|m| m.as_str().starts_with("repeated")), + }) + .collect(); + query_messages.insert(msg_name, fields); + } + + let mut endpoints = Vec::new(); + + for rpc in &rpcs { + // Derive snake_case method name: GetStockHistoryEod → stock_history_eod + let method = rpc_to_method(&rpc.rpc_name); + + // Find the query message: StockHistoryEodRequest → StockHistoryEodRequestQuery + let query_msg_name = format!("{}Query", rpc.request_type); + let fields = if let Some(f) = query_messages.get(&query_msg_name) { + f.clone() + } else { + eprintln!( + "warning: no query message '{}' found, skipping {}", + query_msg_name, rpc.rpc_name + ); + continue; + }; + + // Expand fields (contract_spec → symbol, expiration, strike, right) + let params = expand_fields(&fields); + + let category = derive_category(&method); + let subcategory = derive_subcategory(&method, &category); + let rest_path = derive_rest_path(&method, &category); + let return_type = derive_return_type(&method); + let description = derive_description(&method); + let mut params = params + .into_iter() + .map(|(name, description, param_type, required)| GeneratedParam { + name, + description, + param_type, + required, + binding: String::new(), + arg_name: None, + default: None, + }) + .collect::>(); + normalize_method_params(&method, &mut params); + + endpoints.push(GeneratedEndpoint { + name: method, + description, + category, + subcategory, + rest_path, + grpc_name: format!("get_{}", rpc_to_method(&rpc.rpc_name)), + request_type: rpc.request_type.clone(), + query_type: query_msg_name, + fields, + params, + return_type, + kind: String::new(), + list_column: None, + }); + } + + // ── Manual extra: stock_history_ohlc_range ───────────────────────────── + // Second SDK-level method on top of the same GetStockHistoryOhlc RPC. + // The proto supports both shapes via the optional `date` vs + // `start_date`/`end_date` fields; the SDK exposes them as two distinct + // methods for nicer ergonomics. The registry parser only picks up one + // method per RPC, so the range variant is appended manually here. + endpoints.push(GeneratedEndpoint { + name: "stock_history_ohlc_range".into(), + description: "Fetch intraday OHLC bars across a date range.".into(), + category: "stock".into(), + subcategory: "history".into(), + rest_path: "/v3/stock/history/ohlc_range".into(), + grpc_name: "get_stock_history_ohlc".into(), + request_type: "StockHistoryOhlcRequest".into(), + query_type: "StockHistoryOhlcRequestQuery".into(), + fields: vec![ + ProtoField { + name: "symbol".into(), + proto_type: "string".into(), + is_optional: false, + is_repeated: false, + }, + ProtoField { + name: "date".into(), + proto_type: "string".into(), + is_optional: true, + is_repeated: false, + }, + ProtoField { + name: "interval".into(), + proto_type: "string".into(), + is_optional: false, + is_repeated: false, + }, + ProtoField { + name: "start_time".into(), + proto_type: "string".into(), + is_optional: true, + is_repeated: false, + }, + ProtoField { + name: "end_time".into(), + proto_type: "string".into(), + is_optional: true, + is_repeated: false, + }, + ProtoField { + name: "venue".into(), + proto_type: "string".into(), + is_optional: true, + is_repeated: false, + }, + ProtoField { + name: "start_date".into(), + proto_type: "string".into(), + is_optional: true, + is_repeated: false, + }, + ProtoField { + name: "end_date".into(), + proto_type: "string".into(), + is_optional: true, + is_repeated: false, + }, + ], + params: vec![ + GeneratedParam { + name: "symbol".into(), + description: "Ticker symbol (e.g. AAPL)".into(), + param_type: "Symbol".into(), + required: true, + binding: String::new(), + arg_name: None, + default: None, + }, + GeneratedParam { + name: "start_date".into(), + description: "Start date YYYYMMDD".into(), + param_type: "Date".into(), + required: true, + binding: String::new(), + arg_name: None, + default: None, + }, + GeneratedParam { + name: "end_date".into(), + description: "End date YYYYMMDD".into(), + param_type: "Date".into(), + required: true, + binding: String::new(), + arg_name: None, + default: None, + }, + GeneratedParam { + name: "interval".into(), + description: "Accepts milliseconds (60000) or shorthand (1m). Presets: 100ms, 500ms, 1s, 5s, 10s, 15s, 30s, 1m, 5m, 10m, 15m, 30m, 1h.".into(), + param_type: "Interval".into(), + required: true, + binding: String::new(), + arg_name: None, + default: None, + }, + GeneratedParam { + name: "start_time".into(), + description: "Start time filter (hh:mm:ss.SSS or ms)".into(), + param_type: "Str".into(), + required: false, + binding: String::new(), + arg_name: None, + default: None, + }, + GeneratedParam { + name: "end_time".into(), + description: "End time filter (hh:mm:ss.SSS or ms)".into(), + param_type: "Str".into(), + required: false, + binding: String::new(), + arg_name: None, + default: None, + }, + GeneratedParam { + name: "venue".into(), + description: "Venue/exchange filter".into(), + param_type: "Str".into(), + required: false, + binding: String::new(), + arg_name: None, + default: None, + }, + ], + return_type: "OhlcTicks".into(), + kind: String::new(), + list_column: None, + }); + + Ok(ParsedEndpoints { endpoints }) +} + +/// Load the explicit endpoint surface spec and join it to proto-derived wire metadata. +fn load_endpoint_specs() -> Result> { + let wire = load_proto_endpoints()?; + let spec_path = "endpoint_surface.toml"; + let spec_str = std::fs::read_to_string(spec_path)?; + let spec: SurfaceSpec = toml::from_str(&spec_str)?; + if spec.version != 2 { + return Err(format!( + "unsupported endpoint surface spec version {} in {spec_path}", + spec.version + ) + .into()); + } + + let resolved = resolve_surface_endpoints(&spec)?; + + let mut seen_names = HashSet::new(); + let mut wire_by_name = HashMap::new(); + for endpoint in wire.endpoints { + wire_by_name.insert(endpoint.name.clone(), endpoint); + } + + let mut endpoints = Vec::with_capacity(resolved.len()); + for surface in resolved { + if !seen_names.insert(surface.name.clone()) { + return Err(format!("duplicate endpoint surface entry: {}", surface.name).into()); + } + let wire_name = surface.wire_name.as_deref().unwrap_or(&surface.name); + let wire_endpoint = wire_by_name.get(wire_name).ok_or_else(|| { + format!( + "endpoint surface '{}' references unknown wire endpoint '{}'", + surface.name, wire_name + ) + })?; + + validate_surface_endpoint(&surface, wire_endpoint)?; + endpoints.push(merge_surface_and_wire(surface, wire_endpoint)); + } + + println!("cargo:rerun-if-changed={spec_path}"); + + Ok(ParsedEndpoints { endpoints }) +} + +/// Resolve the reusable spec language in `endpoint_surface.toml` into concrete endpoints. +/// +/// This expands parameter groups, resolves template inheritance, detects +/// cycles, and rejects dead configuration such as unused groups or templates. +fn resolve_surface_endpoints( + spec: &SurfaceSpec, +) -> Result, Box> { + let mut template_cache = HashMap::new(); + let mut param_group_cache = HashMap::new(); + let mut template_stack = Vec::new(); + let mut param_group_stack = Vec::new(); + let mut used_templates = HashSet::new(); + let mut used_param_groups = HashSet::new(); + let mut endpoints = Vec::with_capacity(spec.endpoints.len()); + + for endpoint in &spec.endpoints { + endpoints.push(resolve_surface_endpoint( + endpoint, + spec, + &mut template_cache, + &mut param_group_cache, + &mut template_stack, + &mut param_group_stack, + &mut used_templates, + &mut used_param_groups, + )?); + } + + let mut unused_templates = spec + .templates + .keys() + .filter(|name| !used_templates.contains(*name)) + .cloned() + .collect::>(); + unused_templates.sort(); + if !unused_templates.is_empty() { + return Err(format!( + "unused endpoint templates in endpoint_surface.toml: {}", + unused_templates.join(", ") + ) + .into()); + } + + let mut unused_param_groups = spec + .param_groups + .keys() + .filter(|name| !used_param_groups.contains(*name)) + .cloned() + .collect::>(); + unused_param_groups.sort(); + if !unused_param_groups.is_empty() { + return Err(format!( + "unused parameter groups in endpoint_surface.toml: {}", + unused_param_groups.join(", ") + ) + .into()); + } + + Ok(endpoints) +} + +/// Resolve a single concrete endpoint, applying any referenced template first. +#[allow(clippy::too_many_arguments)] +fn resolve_surface_endpoint( + endpoint: &SurfaceEndpoint, + spec: &SurfaceSpec, + template_cache: &mut HashMap, + param_group_cache: &mut HashMap>, + template_stack: &mut Vec, + param_group_stack: &mut Vec, + used_templates: &mut HashSet, + used_param_groups: &mut HashSet, +) -> Result> { + let template = if let Some(template_name) = endpoint.template.as_deref() { + used_templates.insert(template_name.to_string()); + resolve_surface_template( + template_name, + spec, + template_cache, + param_group_cache, + template_stack, + param_group_stack, + used_templates, + used_param_groups, + )? + } else { + ResolvedTemplate::default() + }; + + let mut params = template.params; + params.extend(resolve_param_entries( + &endpoint.params, + spec, + param_group_cache, + param_group_stack, + used_param_groups, + )?); + + Ok(ResolvedSurfaceEndpoint { + name: endpoint.name.clone(), + wire_name: endpoint.wire_name.clone().or(template.wire_name), + description: resolve_required_surface_field( + endpoint.description.clone().or(template.description), + &endpoint.name, + "description", + )?, + category: resolve_required_surface_field( + endpoint.category.clone().or(template.category), + &endpoint.name, + "category", + )?, + subcategory: resolve_required_surface_field( + endpoint.subcategory.clone().or(template.subcategory), + &endpoint.name, + "subcategory", + )?, + rest_path: resolve_required_surface_field( + endpoint.rest_path.clone().or(template.rest_path), + &endpoint.name, + "rest_path", + )?, + kind: resolve_required_surface_field( + endpoint.kind.clone().or(template.kind), + &endpoint.name, + "kind", + )?, + returns: resolve_required_surface_field( + endpoint.returns.clone().or(template.returns), + &endpoint.name, + "returns", + )?, + list_column: endpoint.list_column.clone().or(template.list_column), + params, + }) +} + +/// Resolve a template, including any inherited parent template chain. +#[allow(clippy::too_many_arguments)] +fn resolve_surface_template( + name: &str, + spec: &SurfaceSpec, + template_cache: &mut HashMap, + param_group_cache: &mut HashMap>, + template_stack: &mut Vec, + param_group_stack: &mut Vec, + used_templates: &mut HashSet, + used_param_groups: &mut HashSet, +) -> Result> { + if let Some(cached) = template_cache.get(name) { + return Ok(cached.clone()); + } + if template_stack.iter().any(|entry| entry == name) { + let mut cycle = template_stack.clone(); + cycle.push(name.to_string()); + return Err(format!("template inheritance cycle: {}", cycle.join(" -> ")).into()); + } + + let template = spec + .templates + .get(name) + .ok_or_else(|| format!("unknown endpoint template '{}'", name))?; + template_stack.push(name.to_string()); + + let mut resolved = if let Some(parent) = template.extends.as_deref() { + used_templates.insert(parent.to_string()); + resolve_surface_template( + parent, + spec, + template_cache, + param_group_cache, + template_stack, + param_group_stack, + used_templates, + used_param_groups, + )? + } else { + ResolvedTemplate::default() + }; + + if let Some(value) = &template.wire_name { + resolved.wire_name = Some(value.clone()); + } + if let Some(value) = &template.description { + resolved.description = Some(value.clone()); + } + if let Some(value) = &template.category { + resolved.category = Some(value.clone()); + } + if let Some(value) = &template.subcategory { + resolved.subcategory = Some(value.clone()); + } + if let Some(value) = &template.rest_path { + resolved.rest_path = Some(value.clone()); + } + if let Some(value) = &template.kind { + resolved.kind = Some(value.clone()); + } + if let Some(value) = &template.returns { + resolved.returns = Some(value.clone()); + } + if let Some(value) = &template.list_column { + resolved.list_column = Some(value.clone()); + } + resolved.params.extend(resolve_param_entries( + &template.params, + spec, + param_group_cache, + param_group_stack, + used_param_groups, + )?); + + template_stack.pop(); + template_cache.insert(name.to_string(), resolved.clone()); + Ok(resolved) +} + +/// Expand a sequence of parameter entries, recursively resolving group references. +fn resolve_param_entries( + entries: &[SurfaceParamEntry], + spec: &SurfaceSpec, + param_group_cache: &mut HashMap>, + param_group_stack: &mut Vec, + used_param_groups: &mut HashSet, +) -> Result, Box> { + let mut params = Vec::new(); + for entry in entries { + match entry { + SurfaceParamEntry::Param(param) => params.push(param.clone()), + SurfaceParamEntry::Use(param_use) => { + used_param_groups.insert(param_use.group.clone()); + params.extend(resolve_param_group( + ¶m_use.group, + spec, + param_group_cache, + param_group_stack, + used_param_groups, + )?); + } + } + } + Ok(params) +} + +/// Resolve a reusable parameter group with cycle detection and memoization. +fn resolve_param_group( + name: &str, + spec: &SurfaceSpec, + param_group_cache: &mut HashMap>, + param_group_stack: &mut Vec, + used_param_groups: &mut HashSet, +) -> Result, Box> { + if let Some(cached) = param_group_cache.get(name) { + return Ok(cached.clone()); + } + if param_group_stack.iter().any(|entry| entry == name) { + let mut cycle = param_group_stack.clone(); + cycle.push(name.to_string()); + return Err(format!("parameter group cycle: {}", cycle.join(" -> ")).into()); + } + + let group = spec + .param_groups + .get(name) + .ok_or_else(|| format!("unknown parameter group '{}'", name))?; + param_group_stack.push(name.to_string()); + let params = resolve_param_entries( + &group.params, + spec, + param_group_cache, + param_group_stack, + used_param_groups, + )?; + param_group_stack.pop(); + param_group_cache.insert(name.to_string(), params.clone()); + Ok(params) +} + +/// Require a fully-resolved endpoint field after template inheritance has been applied. +fn resolve_required_surface_field( + value: Option, + endpoint_name: &str, + field_name: &str, +) -> Result> { + value.ok_or_else(|| { + format!( + "endpoint '{}' is missing required field '{}'", + endpoint_name, field_name + ) + .into() + }) +} + +fn validate_surface_endpoint( + surface: &ResolvedSurfaceEndpoint, + wire: &GeneratedEndpoint, +) -> Result<(), Box> { + match surface.kind.as_str() { + "list" | "parsed" => {} + other => { + return Err(format!( + "endpoint '{}' has unsupported kind '{}'", + surface.name, other + ) + .into()) + } + } + + if surface.kind == "list" && surface.returns != "StringList" { + return Err(format!( + "list endpoint '{}' must return StringList, got {}", + surface.name, surface.returns + ) + .into()); + } + if surface.kind != "list" && surface.list_column.is_some() { + return Err(format!( + "parsed endpoint '{}' cannot define list_column", + surface.name + ) + .into()); + } + if surface.kind == "list" && surface.list_column.is_none() { + return Err(format!("list endpoint '{}' must define list_column", surface.name).into()); + } + if surface.returns != wire.return_type { + return Err(format!( + "endpoint '{}' declares return type {} but wire-derived model uses {}", + surface.name, surface.returns, wire.return_type + ) + .into()); + } + + let wire_params = wire + .params + .iter() + .map(|param| (param.name.as_str(), param)) + .collect::>(); + let mut surface_names = HashSet::new(); + for param in &surface.params { + if !surface_names.insert(param.name.clone()) { + return Err(format!( + "endpoint '{}' defines duplicate param '{}'", + surface.name, param.name + ) + .into()); + } + let wire_param = wire_params.get(param.name.as_str()).ok_or_else(|| { + format!( + "endpoint '{}' declares param '{}' not present in wire endpoint '{}'", + surface.name, param.name, wire.name + ) + })?; + if param.param_type != wire_param.param_type { + return Err(format!( + "endpoint '{}.{}' declares type {} but wire-derived model uses {}", + surface.name, param.name, param.param_type, wire_param.param_type + ) + .into()); + } + if wire_param.required && !param.required { + return Err(format!( + "endpoint '{}.{}' relaxes a required wire parameter", + surface.name, param.name + ) + .into()); + } + match param.binding.as_str() { + "method" | "builder" => {} + other => { + return Err(format!( + "endpoint '{}.{}' has unsupported binding '{}'", + surface.name, param.name, other + ) + .into()) + } + } + if param.required && param.default.is_some() { + return Err(format!( + "endpoint '{}.{}' cannot define a default for a required parameter", + surface.name, param.name + ) + .into()); + } + if param.binding == "method" && !param.required { + return Err(format!( + "endpoint '{}.{}' cannot declare an optional method-bound parameter", + surface.name, param.name + ) + .into()); + } + if param.default.is_some() && param.binding != "builder" { + return Err(format!( + "endpoint '{}.{}' can only define defaults for builder-bound parameters", + surface.name, param.name + ) + .into()); + } + } + + for wire_param in &wire.params { + let missing_from_surface = !surface_names.contains(&wire_param.name); + let must_be_present = surface.wire_name.is_none() || wire_param.required; + if missing_from_surface && must_be_present { + return Err(format!( + "endpoint '{}' is missing wire parameter '{}' in endpoint_surface.toml", + surface.name, wire_param.name + ) + .into()); + } + } + + Ok(()) +} + +fn merge_surface_and_wire( + surface: ResolvedSurfaceEndpoint, + wire: &GeneratedEndpoint, +) -> GeneratedEndpoint { + GeneratedEndpoint { + name: surface.name, + description: surface.description, + category: surface.category, + subcategory: surface.subcategory, + rest_path: surface.rest_path, + grpc_name: wire.grpc_name.clone(), + request_type: wire.request_type.clone(), + query_type: wire.query_type.clone(), + fields: wire.fields.clone(), + params: surface + .params + .into_iter() + .map(|param| GeneratedParam { + name: param.name, + description: param.description, + param_type: param.param_type, + required: param.required, + binding: param.binding, + arg_name: param.arg_name, + default: param.default, + }) + .collect(), + return_type: surface.returns, + kind: surface.kind, + list_column: surface.list_column, + } +} + +pub fn generate_all() -> Result<(), Box> { + generate_endpoint_registry()?; + generate_endpoint_runtime()?; + generate_direct_endpoints()?; + println!("cargo:rerun-if-changed=proto/external.proto"); + Ok(()) +} + +fn generate_endpoint_registry() -> Result<(), Box> { + let parsed = load_endpoint_specs()?; + + // ── Generate Rust code ────────────────────────────────────────────────── + let mut code = String::new(); + code.push_str( + "// Auto-generated by build.rs from endpoint_surface.toml validated against external.proto.\n", + ); + code.push_str("// Do not edit manually.\n\n"); + code.push_str("pub static ENDPOINTS: &[EndpointMeta] = &[\n"); + + for endpoint in &parsed.endpoints { + code.push_str(" EndpointMeta {\n"); + writeln!(code, " name: \"{}\",", endpoint.name).unwrap(); + writeln!(code, " description: \"{}\",", endpoint.description).unwrap(); + writeln!(code, " category: \"{}\",", endpoint.category).unwrap(); + writeln!(code, " subcategory: \"{}\",", endpoint.subcategory).unwrap(); + writeln!(code, " rest_path: \"{}\",", endpoint.rest_path).unwrap(); + + if endpoint.params.is_empty() { + code.push_str(" params: &[],\n"); + } else { + code.push_str(" params: &[\n"); + for param in &endpoint.params { + code.push_str(" ParamMeta {\n"); + writeln!(code, " name: \"{}\",", param.name).unwrap(); + writeln!( + code, + " description: \"{}\",", + param.description + ) + .unwrap(); + writeln!( + code, + " param_type: ParamType::{},", + param.param_type + ) + .unwrap(); + writeln!(code, " required: {},", param.required).unwrap(); + code.push_str(" },\n"); + } + code.push_str(" ],\n"); + } + + writeln!( + code, + " returns: ReturnType::{},", + endpoint.return_type + ) + .unwrap(); + code.push_str(" },\n"); + } + + code.push_str("];\n"); + + let out_dir = std::env::var("OUT_DIR")?; + let dest = Path::new(&out_dir).join("registry_generated.rs"); + std::fs::write(&dest, &code)?; + + Ok(()) +} + +fn generate_endpoint_runtime() -> Result<(), Box> { + let parsed = load_endpoint_specs()?; + + let mut code = String::new(); + code.push_str( + "// Auto-generated by build.rs from endpoint_surface.toml validated against external.proto.\n", + ); + code.push_str("// Do not edit manually.\n\n"); + code.push_str("/// Dispatch a validated endpoint call into the generated SDK adapter set.\n"); + code.push_str("///\n"); + code.push_str( + "/// The build script emits one match arm per endpoint from the shared endpoint\n", + ); + code.push_str( + "/// metadata so registry-driven projections stay aligned with the SDK surface.\n", + ); + code.push_str("pub async fn invoke_generated_endpoint(\n"); + code.push_str(" client: &crate::ThetaDataDx,\n"); + code.push_str(" name: &str,\n"); + code.push_str(" args: &EndpointArgs,\n"); + code.push_str(") -> Result {\n"); + code.push_str(" match name {\n"); + + for endpoint in &parsed.endpoints { + generate_endpoint_dispatch_arm(&mut code, endpoint); + } + + code.push_str(" _ => Err(EndpointError::UnknownEndpoint(name.to_string())),\n"); + code.push_str(" }\n"); + code.push_str("}\n"); + + let out_dir = std::env::var("OUT_DIR")?; + let dest = Path::new(&out_dir).join("endpoint_generated.rs"); + std::fs::write(&dest, &code)?; + + Ok(()) +} + +fn generate_direct_endpoints() -> Result<(), Box> { + let parsed = load_endpoint_specs()?; + + let mut list_code = String::new(); + let mut parsed_code = String::new(); + list_code.push_str( + "// Auto-generated by build.rs from endpoint_surface.toml validated against external.proto.\n", + ); + list_code.push_str("// Do not edit manually.\n\n"); + list_code.push_str("impl DirectClient {\n"); + parsed_code.push_str( + "// Auto-generated by build.rs from endpoint_surface.toml validated against external.proto.\n", + ); + parsed_code.push_str("// Do not edit manually.\n\n"); + + for endpoint in &parsed.endpoints { + if is_simple_list_endpoint(endpoint) { + generate_direct_list_endpoint(&mut list_code, endpoint); + } else { + generate_direct_parsed_endpoint(&mut parsed_code, endpoint); + } + } + + let out_dir = std::env::var("OUT_DIR")?; + std::fs::write( + Path::new(&out_dir).join("direct_list_endpoints_generated.rs"), + format!("{list_code}}}\n"), + )?; + std::fs::write( + Path::new(&out_dir).join("direct_parsed_endpoints_generated.rs"), + parsed_code, + )?; + + Ok(()) +} + +fn generate_direct_list_endpoint(out: &mut String, endpoint: &GeneratedEndpoint) { + writeln!(out, "list_endpoint! {{").unwrap(); + writeln!(out, " #[doc = {:?}]", endpoint.description).unwrap(); + writeln!( + out, + " #[doc = {:?}]", + format!("gRPC stub: `{}`", endpoint.grpc_name) + ) + .unwrap(); + + let signature = endpoint + .params + .iter() + .map(|param| format!("{}: &str", direct_method_arg_name(endpoint, param))) + .collect::>() + .join(", "); + if signature.is_empty() { + writeln!( + out, + " fn {}() -> {:?};", + endpoint.name, + endpoint + .list_column + .as_deref() + .expect("list endpoint must declare list_column") + ) + .unwrap(); + } else { + writeln!( + out, + " fn {}({signature}) -> {:?};", + endpoint.name, + endpoint + .list_column + .as_deref() + .expect("list endpoint must declare list_column") + ) + .unwrap(); + } + + writeln!(out, " grpc: {};", endpoint.grpc_name).unwrap(); + writeln!(out, " request: {};", endpoint.request_type).unwrap(); + if endpoint.fields.is_empty() { + writeln!(out, " query: {} {{}};", endpoint.query_type).unwrap(); + } else { + writeln!(out, " query: {} {{", endpoint.query_type).unwrap(); + for field in &endpoint.fields { + writeln!( + out, + " {}: {},", + field.name, + direct_query_field_expr(endpoint, field, true) + ) + .unwrap(); + } + out.push_str(" };\n"); + } + out.push_str("}\n\n"); +} + +fn generate_direct_parsed_endpoint(out: &mut String, endpoint: &GeneratedEndpoint) { + writeln!(out, "parsed_endpoint! {{").unwrap(); + writeln!(out, " #[doc = {:?}]", endpoint.description).unwrap(); + writeln!( + out, + " #[doc = {:?}]", + format!("gRPC stub: `{}`", endpoint.grpc_name) + ) + .unwrap(); + writeln!( + out, + " builder {}Builder;", + to_pascal_case(&endpoint.name) + ) + .unwrap(); + + let method_params = endpoint + .params + .iter() + .filter(|param| is_method_call_param(param)) + .collect::>(); + let signature = method_params + .iter() + .map(|param| { + format!( + "{}: {}", + direct_method_arg_name(endpoint, param), + direct_required_kind(param) + ) + }) + .collect::>() + .join(", "); + writeln!( + out, + " fn {}({signature}) -> {};", + endpoint.name, + direct_return_type(&endpoint.return_type) + ) + .unwrap(); + + writeln!(out, " grpc: {};", endpoint.grpc_name).unwrap(); + writeln!(out, " request: {};", endpoint.request_type).unwrap(); + if endpoint.fields.is_empty() { + writeln!(out, " query: {} {{}};", endpoint.query_type).unwrap(); + } else { + writeln!(out, " query: {} {{", endpoint.query_type).unwrap(); + for field in &endpoint.fields { + writeln!( + out, + " {}: {},", + field.name, + direct_query_field_expr(endpoint, field, false) + ) + .unwrap(); + } + out.push_str(" };\n"); + } + writeln!( + out, + " parse: {};", + direct_parser_name(&endpoint.return_type) + ) + .unwrap(); + + let date_args = method_params + .iter() + .filter_map(|param| direct_date_arg_name(endpoint, param)) + .collect::>(); + if !date_args.is_empty() { + writeln!(out, " dates: {};", date_args.join(", ")).unwrap(); + } + + let optional_params = endpoint + .params + .iter() + .filter(|param| !is_method_call_param(param)) + .collect::>(); + if optional_params.is_empty() { + out.push_str(" optional {}\n"); + } else { + out.push_str(" optional {\n"); + for param in optional_params { + let (kind, default) = direct_optional_kind_and_default(param); + writeln!(out, " {}: {} = {},", param.name, kind, default).unwrap(); + } + out.push_str(" }\n"); + } + out.push_str("}\n\n"); +} + +fn direct_method_arg_name(endpoint: &GeneratedEndpoint, param: &GeneratedParam) -> String { + let _ = endpoint; + param.arg_name.clone().unwrap_or_else(|| param.name.clone()) +} + +fn direct_date_arg_name(endpoint: &GeneratedEndpoint, param: &GeneratedParam) -> Option { + match param.name.as_str() { + "date" | "start_date" | "end_date" => Some(direct_method_arg_name(endpoint, param)), + _ => None, + } +} + +fn direct_required_kind(param: &GeneratedParam) -> &'static str { + if param.param_type == "Symbols" { + "str_vec" + } else { + "str" + } +} + +fn direct_optional_kind_and_default(param: &GeneratedParam) -> (&'static str, String) { + if let Some(default) = param.default.as_deref() { + return match param.param_type.as_str() { + "Str" => ("string", format!("{default:?}.to_string()")), + "Int" => { + let value = default.parse::().unwrap_or_else(|_| { + panic!( + "invalid int default '{}' for parameter '{}'", + default, param.name + ) + }); + ("opt_i32", format!("Some({value})")) + } + "Float" => { + let value = default.parse::().unwrap_or_else(|_| { + panic!( + "invalid float default '{}' for parameter '{}'", + default, param.name + ) + }); + ("opt_f64", format!("Some({value:?})")) + } + "Bool" => { + let value = default.parse::().unwrap_or_else(|_| { + panic!( + "invalid bool default '{}' for parameter '{}'", + default, param.name + ) + }); + ("opt_bool", format!("Some({value})")) + } + other => panic!( + "unsupported default for parameter '{}' with type '{}'", + param.name, other + ), + }; + } + match param.param_type.as_str() { + "Int" => ("opt_i32", "None".into()), + "Float" => ("opt_f64", "None".into()), + "Bool" => ("opt_bool", "None".into()), + _ => ("opt_str", "None".into()), + } +} + +fn is_simple_list_endpoint(endpoint: &GeneratedEndpoint) -> bool { + endpoint.kind == "list" +} + +fn is_method_call_param(param: &GeneratedParam) -> bool { + param.binding == "method" +} + +fn required_getter_name(param_type: &str) -> &'static str { + match param_type { + "Symbol" => "required_symbol", + "Symbols" => "required_symbols", + "Date" | "Expiration" => "required_date", + "Interval" => "required_interval", + "Right" => "required_right", + "Int" => "required_int32", + "Float" => "required_float64", + "Bool" => "required_bool", + "Year" => "required_year", + _ => "required_str", + } +} + +fn optional_getter_name(param_type: &str) -> &'static str { + match param_type { + "Date" | "Expiration" => "optional_date", + "Int" => "optional_int32", + "Float" => "optional_float64", + "Bool" => "optional_bool", + _ => "optional_str", + } +} + +fn is_symbols_param(param: &GeneratedParam) -> bool { + param.param_type == "Symbols" +} + +fn call_arg_name(param: &GeneratedParam) -> String { + if is_symbols_param(param) { + "&symbol_refs".into() + } else { + param.name.clone() + } +} + +fn direct_return_type(return_type: &str) -> &'static str { + match return_type { + "EodTicks" => "Vec", + "OhlcTicks" => "Vec", + "TradeTicks" => "Vec", + "QuoteTicks" => "Vec", + "TradeQuoteTicks" => "Vec", + "OpenInterestTicks" => "Vec", + "MarketValueTicks" => "Vec", + "GreeksTicks" => "Vec", + "IvTicks" => "Vec", + "PriceTicks" => "Vec", + "CalendarDays" => "Vec", + "InterestRateTicks" => "Vec", + "OptionContracts" => "Vec", + other => panic!("unsupported direct return type: {other}"), + } +} + +fn direct_parser_name(return_type: &str) -> &'static str { + match return_type { + "EodTicks" => "decode::parse_eod_ticks", + "OhlcTicks" => "decode::parse_ohlc_ticks", + "TradeTicks" => "decode::parse_trade_ticks", + "QuoteTicks" => "decode::parse_quote_ticks", + "TradeQuoteTicks" => "decode::parse_trade_quote_ticks", + "OpenInterestTicks" => "decode::parse_open_interest_ticks", + "MarketValueTicks" => "decode::parse_market_value_ticks", + "GreeksTicks" => "decode::parse_greeks_ticks", + "IvTicks" => "decode::parse_iv_ticks", + "PriceTicks" => "decode::parse_price_ticks", + "CalendarDays" => "decode::parse_calendar_days_v3", + "InterestRateTicks" => "decode::parse_interest_rate_ticks", + "OptionContracts" => "decode::parse_option_contracts_v3", + other => panic!("unsupported parser return type: {other}"), + } +} + +fn direct_query_field_expr( + endpoint: &GeneratedEndpoint, + field: &ProtoField, + list_context: bool, +) -> String { + if field.proto_type == "ContractSpec" { + return "contract_spec!(symbol, expiration, strike, right)".into(); + } + if field.name == "date" && endpoint.name == "stock_history_ohlc_range" { + return "None".into(); + } + + let param = endpoint + .params + .iter() + .find(|param| param.name == field.name) + .unwrap_or_else(|| { + panic!( + "missing param metadata for {}::{}", + endpoint.name, field.name + ) + }); + let arg_name = direct_method_arg_name(endpoint, param); + let is_method_param = is_method_call_param(param); + + match field.name.as_str() { + "symbol" if field.is_repeated => { + if param.param_type == "Symbols" { + "symbols.clone()".into() + } else if list_context { + format!("vec![{arg_name}.to_string()]") + } else { + format!("vec![{arg_name}.clone()]") + } + } + "interval" => format!("normalize_interval(&{arg_name})"), + "start_time" | "end_time" => format!("Some({arg_name}.clone())"), + "venue" if endpoint.category == "stock" => { + "venue.clone().or_else(|| Some(\"nqb\".to_string()))".into() + } + _ if field.proto_type == "string" => { + if field.is_optional { + if is_method_param { + format!("Some({arg_name}.clone())") + } else { + format!("{arg_name}.clone()") + } + } else if list_context { + format!("{arg_name}.to_string()") + } else { + format!("{arg_name}.clone()") + } + } + _ => arg_name, + } +} + +fn to_pascal_case(value: &str) -> String { + value + .split('_') + .filter(|segment| !segment.is_empty()) + .map(|segment| { + let mut chars = segment.chars(); + match chars.next() { + Some(first) => first.to_uppercase().to_string() + chars.as_str(), + None => String::new(), + } + }) + .collect::() +} + +fn generate_endpoint_dispatch_arm(out: &mut String, endpoint: &GeneratedEndpoint) { + writeln!(out, " \"{}\" => {{", endpoint.name).unwrap(); + + if is_simple_list_endpoint(endpoint) { + for param in &endpoint.params { + emit_required_arg(out, endpoint, param); + } + let args = endpoint + .params + .iter() + .map(call_arg_name) + .collect::>() + .join(", "); + writeln!( + out, + " let values = client.{}({args}).await?;", + endpoint.name + ) + .unwrap(); + writeln!( + out, + " Ok(EndpointOutput::{}(values))", + endpoint.return_type + ) + .unwrap(); + out.push_str(" }\n"); + return; + } + + let method_call_params = endpoint + .params + .iter() + .filter(|param| is_method_call_param(param)) + .collect::>(); + let builder_params = endpoint + .params + .iter() + .filter(|param| !is_method_call_param(param)) + .collect::>(); + + for param in &method_call_params { + emit_required_arg(out, endpoint, param); + } + + let call_args = method_call_params + .into_iter() + .map(call_arg_name) + .collect::>() + .join(", "); + + if builder_params.is_empty() { + writeln!( + out, + " let result = client.{}({call_args}).await?;", + endpoint.name + ) + .unwrap(); + } else { + writeln!( + out, + " let mut builder = client.{}({call_args});", + endpoint.name + ) + .unwrap(); + + for param in builder_params { + let getter = optional_getter_name(¶m.param_type); + writeln!( + out, + " if let Some(value) = args.{getter}(\"{}\")? {{", + param.name + ) + .unwrap(); + writeln!( + out, + " builder = builder.{}(value);", + param.name + ) + .unwrap(); + out.push_str(" }\n"); + } + + out.push_str(" let result = builder.await?;\n"); + } + writeln!( + out, + " Ok(EndpointOutput::{}(result))", + endpoint.return_type + ) + .unwrap(); + out.push_str(" }\n"); +} + +fn emit_required_arg(out: &mut String, _endpoint: &GeneratedEndpoint, param: &GeneratedParam) { + if param.param_type == "Symbols" { + writeln!( + out, + " let symbol_values = args.required_symbols(\"{}\")?;", + param.name + ) + .unwrap(); + out.push_str( + " let symbol_refs: Vec<&str> = symbol_values.iter().map(String::as_str).collect();\n", + ); + return; + } + + let getter = required_getter_name(¶m.param_type); + writeln!( + out, + " let {} = args.{getter}(\"{}\")?;", + param.name, param.name + ) + .unwrap(); +} + +fn normalize_method_params(method: &str, params: &mut [GeneratedParam]) { + let supports_symbol_lists = + method.starts_with("stock_snapshot_") || method.starts_with("index_snapshot_"); + + if !supports_symbol_lists { + for param in params.iter_mut() { + if param.name == "symbol" && param.param_type == "Symbols" { + param.param_type = "Symbol".into(); + param.description = "Ticker symbol (e.g. AAPL)".into(); + } + } + } +} + +fn is_simple_list_method(method: &str) -> bool { + method.ends_with("_list_symbols") + || method.ends_with("_list_dates") + || method.ends_with("_list_expirations") + || method.ends_with("_list_strikes") +} + +/// Convert `GetStockHistoryEod` → `stock_history_eod`. +fn rpc_to_method(rpc_name: &str) -> String { + // Strip leading "Get" + let name = rpc_name.strip_prefix("Get").unwrap_or(rpc_name); // build script: panic is intentional + // PascalCase → snake_case + let mut result = String::new(); + for (i, ch) in name.chars().enumerate() { + if ch.is_uppercase() { + if i > 0 { + result.push('_'); + } + result.push(ch.to_lowercase().next().unwrap()); // build script: panic is intentional + } else { + result.push(ch); + } + } + result +} + +/// Expand proto fields, replacing `contract_spec` with (symbol, expiration, strike, right). +/// +/// Many option query messages carry both a `ContractSpec` (contract identity, +/// expanded here to 4 fields) AND an explicit top-level `expiration` field +/// (the query range expiration — e.g. "include all contracts expiring by..."), +/// which would otherwise collide with the contract's own expiration. Any +/// post-expansion duplicate parameter name is dropped in favor of the first +/// occurrence (ContractSpec wins, since it is structurally the contract +/// identity the user really cares about). +fn expand_fields(fields: &[ProtoField]) -> Vec<(String, String, String, bool)> { + let mut params: Vec<(String, String, String, bool)> = Vec::new(); + let mut seen: std::collections::HashSet = std::collections::HashSet::new(); + + let push = |params: &mut Vec<(String, String, String, bool)>, + seen: &mut std::collections::HashSet, + entry: (String, String, String, bool)| { + if seen.insert(entry.0.clone()) { + params.push(entry); + } + }; + + for f in fields { + if f.proto_type == "ContractSpec" { + // Expand to the 4 contract spec fields (symbol, expiration, strike, right). + push( + &mut params, + &mut seen, + ( + "symbol".into(), + "Underlying symbol (e.g. AAPL)".into(), + "Symbol".into(), + true, + ), + ); + push( + &mut params, + &mut seen, + ( + "expiration".into(), + "Expiration date YYYYMMDD".into(), + "Expiration".into(), + true, + ), + ); + push( + &mut params, + &mut seen, + ( + "strike".into(), + "Strike price (raw integer)".into(), + "Strike".into(), + true, + ), + ); + push( + &mut params, + &mut seen, + ( + "right".into(), + "C for call, P for put".into(), + "Right".into(), + true, + ), + ); + } else { + let (param_type, desc) = map_field(&f.name, &f.proto_type, f.is_repeated); + let required = !f.is_optional; + push( + &mut params, + &mut seen, + (f.name.clone(), desc, param_type, required), + ); + } + } + params +} + +/// Map a proto field (name + type + repeated) to (`ParamType` variant name, description). +fn map_field(name: &str, proto_type: &str, is_repeated: bool) -> (String, String) { + // Repeated string symbol → Symbols + if is_repeated && name == "symbol" { + return ( + "Symbols".into(), + "Comma-separated ticker symbols (e.g. AAPL,MSFT)".into(), + ); + } + + match (proto_type, name) { + ("string", "symbol") => ("Symbol".into(), "Ticker symbol (e.g. AAPL)".into()), + ("string", "start_date") => ("Date".into(), "Start date YYYYMMDD".into()), + ("string", "end_date") => ("Date".into(), "End date YYYYMMDD".into()), + ("string", "date") => ("Date".into(), "Date YYYYMMDD".into()), + ("string", "interval") => ( + "Interval".into(), + "Accepts milliseconds (60000) or shorthand (1m). Presets: 100ms, 500ms, 1s, 5s, 10s, 15s, 30s, 1m, 5m, 10m, 15m, 30m, 1h.".into(), + ), + ("string", "right") => ("Right".into(), "C for call, P for put".into()), + ("string", "strike") => ("Strike".into(), "Strike price (raw integer)".into()), + ("string", "expiration") => ("Expiration".into(), "Expiration date YYYYMMDD".into()), + ("string", "request_type") => ( + "RequestType".into(), + "Request type: EOD, TRADE, QUOTE, OHLC, etc.".into(), + ), + ("string", "year") => ("Year".into(), "4-digit year (e.g. 2024)".into()), + ("string", "time_of_day") => ( + "Str".into(), + "Milliseconds from midnight ET (e.g. 34200000 = 9:30 AM)".into(), + ), + ("string", "venue") => ("Str".into(), "Venue/exchange filter".into()), + ("string", "min_time") => ("Str".into(), "Minimum time filter".into()), + ("string", "start_time") => ("Str".into(), "Start time filter".into()), + ("string", "end_time") => ("Str".into(), "End time filter".into()), + ("string", "rate_type") => ("Str".into(), "Rate type".into()), + ("string", "version") => ("Str".into(), "Greeks model version".into()), + ("double", _) => ("Float".into(), humanize_name(name).clone()), + ("int32", "max_dte") => ("Int".into(), "Maximum days to expiration".into()), + ("int32", "strike_range") => ("Int".into(), "Strike range filter".into()), + ("int32", _) => ("Int".into(), humanize_name(name).clone()), + ("bool", "exclusive") => ("Bool".into(), "Exclusive time boundary".into()), + ("bool", "use_market_value") => ("Bool".into(), "Use market value for Greeks".into()), + ("bool", "underlyer_use_nbbo") => ("Bool".into(), "Use NBBO for underlyer price".into()), + ("bool", _) => ("Bool".into(), humanize_name(name).clone()), + _ => ("Str".into(), humanize_name(name).clone()), + } +} + +fn humanize_name(name: &str) -> String { + name.replace('_', " ") + .split_whitespace() + .enumerate() + .map(|(i, w)| { + if i == 0 { + let mut c = w.chars(); + match c.next() { + Some(first) => first.to_uppercase().to_string() + c.as_str(), + None => String::new(), + } + } else { + w.to_string() + } + }) + .collect::>() + .join(" ") +} + +fn derive_category(method: &str) -> String { + if method.starts_with("stock_") { + "stock".into() + } else if method.starts_with("option_") { + "option".into() + } else if method.starts_with("index_") { + "index".into() + } else if method.starts_with("calendar_") { + "calendar".into() + } else if method.starts_with("interest_rate_") { + "rate".into() + } else { + "other".into() + } +} + +fn derive_subcategory(method: &str, category: &str) -> String { + // Strip the category prefix to get the rest + let rest = match category { + // build script: panic is intentional (unwrap_or provides safe fallback) + "stock" => method.strip_prefix("stock_").unwrap_or(method), + "option" => method.strip_prefix("option_").unwrap_or(method), + "index" => method.strip_prefix("index_").unwrap_or(method), + "calendar" => method.strip_prefix("calendar_").unwrap_or(method), + "rate" => method.strip_prefix("interest_rate_").unwrap_or(method), + _ => method, + }; + + if rest.starts_with("list_") { + "list".into() + } else if rest.starts_with("snapshot_greeks_") || rest.starts_with("snapshot_greeks") { + "snapshot_greeks".into() + } else if rest.starts_with("snapshot_") { + "snapshot".into() + } else if rest.starts_with("history_trade_greeks_") { + "history_trade_greeks".into() + } else if rest.starts_with("history_greeks_") { + "history_greeks".into() + } else if rest.starts_with("history_") { + "history".into() + } else if rest.starts_with("at_time_") { + "at_time".into() + } else if rest == "open_today" { + "status".into() + } else if rest == "on_date" || rest == "year" { + "query".into() + } else { + "other".into() + } +} + +fn derive_rest_path(method: &str, category: &str) -> String { + let rest = match category { + "rate" => method.strip_prefix("interest_rate_").unwrap_or(method), + _ => method + .strip_prefix(&format!("{category}_")) + .unwrap_or(method), + }; + + let path_rest = if let Some(what) = rest.strip_prefix("history_trade_greeks_") { + format!("history/trade_greeks/{what}") + } else if let Some(what) = rest.strip_prefix("history_greeks_") { + format!("history/greeks/{what}") + } else if let Some(what) = rest.strip_prefix("snapshot_greeks_") { + format!("snapshot/greeks/{what}") + } else if let Some(what) = rest.strip_prefix("history_") { + format!("history/{what}") + } else if let Some(what) = rest.strip_prefix("snapshot_") { + format!("snapshot/{what}") + } else if let Some(what) = rest.strip_prefix("list_") { + format!("list/{what}") + } else if let Some(what) = rest.strip_prefix("at_time_") { + format!("at_time/{what}") + } else { + rest.to_string() + }; + + format!("/v3/{category}/{path_rest}") +} + +fn derive_return_type(method: &str) -> String { + if is_simple_list_method(method) { + return "StringList".into(); + } + + if method == "option_list_contracts" { + return "OptionContracts".into(); + } + + if method.starts_with("calendar_") { + return "CalendarDays".into(); + } + + if method.starts_with("interest_rate_") { + return "InterestRateTicks".into(); + } + + if method.contains("_trade_quote") { + return "TradeQuoteTicks".into(); + } + + if method.contains("_open_interest") { + return "OpenInterestTicks".into(); + } + + if method.contains("_market_value") { + return "MarketValueTicks".into(); + } + + if method.contains("greeks_implied_volatility") { + return "IvTicks".into(); + } + + if method.contains("_greeks_") { + return "GreeksTicks".into(); + } + + if method == "index_snapshot_price" + || method == "index_history_price" + || method == "index_at_time_price" + { + return "PriceTicks".into(); + } + + if method.ends_with("_history_eod") { + return "EodTicks".into(); + } + + if method.contains("_ohlc") { + return "OhlcTicks".into(); + } + + if method.contains("_trade") || method.ends_with("_trade") { + return "TradeTicks".into(); + } + + if method.contains("_quote") || method.ends_with("_quote") { + return "QuoteTicks".into(); + } + + panic!("unhandled return type mapping for endpoint {method}"); +} + +// Reason: one match dispatch per endpoint — cannot be meaningfully split. +#[allow(clippy::too_many_lines)] +fn derive_description(method: &str) -> String { + // Hand-crafted descriptions for known patterns + match method { + // Stock List + "stock_list_symbols" => "List all available stock ticker symbols.".into(), + "stock_list_dates" => { + "List available dates for a stock by request type (EOD, TRADE, QUOTE, etc.).".into() + } + // Stock Snapshot + "stock_snapshot_ohlc" => "Get the latest OHLC snapshot for one or more stocks.".into(), + "stock_snapshot_trade" => "Get the latest trade snapshot for one or more stocks.".into(), + "stock_snapshot_quote" => { + "Get the latest NBBO quote snapshot for one or more stocks.".into() + } + "stock_snapshot_market_value" => { + "Get the latest market value snapshot for one or more stocks.".into() + } + // Stock History + "stock_history_eod" => { + "Fetch end-of-day stock data for a date range. Returns OHLCV + bid/ask per trading day." + .into() + } + "stock_history_ohlc" => "Fetch intraday OHLC bars for a stock on a single date.".into(), + "stock_history_trade" => "Fetch all trades for a stock on a given date.".into(), + "stock_history_quote" => { + "Fetch NBBO quotes for a stock on a given date at a given interval.".into() + } + "stock_history_trade_quote" => { + "Fetch combined trade + quote ticks for a stock on a given date. Returns raw DataTable." + .into() + } + // Stock At-Time + "stock_at_time_trade" => { + "Fetch the trade at a specific time of day across a date range.".into() + } + "stock_at_time_quote" => { + "Fetch the quote at a specific time of day across a date range.".into() + } + // Option List + "option_list_symbols" => "List all available option underlying symbols.".into(), + "option_list_dates" => { + "List available dates for an option contract by request type.".into() + } + "option_list_expirations" => { + "List available expiration dates for an option underlying.".into() + } + "option_list_strikes" => { + "List available strike prices for an option at a given expiration.".into() + } + "option_list_contracts" => "List all option contracts for a symbol on a given date.".into(), + // Option Snapshot + "option_snapshot_ohlc" => "Get the latest OHLC snapshot for an option contract.".into(), + "option_snapshot_trade" => "Get the latest trade snapshot for an option contract.".into(), + "option_snapshot_quote" => { + "Get the latest NBBO quote snapshot for an option contract.".into() + } + "option_snapshot_open_interest" => { + "Get the latest open interest snapshot for an option contract.".into() + } + "option_snapshot_market_value" => { + "Get the latest market value snapshot for an option contract.".into() + } + // Option Snapshot Greeks + "option_snapshot_greeks_implied_volatility" => { + "Get implied volatility snapshot for an option contract (from ThetaData server).".into() + } + "option_snapshot_greeks_all" => { + "Get all Greeks snapshot for an option contract (from ThetaData server).".into() + } + "option_snapshot_greeks_first_order" => { + "Get first-order Greeks snapshot (delta, theta, rho) for an option contract.".into() + } + "option_snapshot_greeks_second_order" => { + "Get second-order Greeks snapshot (gamma, vanna, charm) for an option contract.".into() + } + "option_snapshot_greeks_third_order" => { + "Get third-order Greeks snapshot (speed, color, ultima) for an option contract.".into() + } + // Option History + "option_history_eod" => { + "Fetch end-of-day option data for a contract over a date range.".into() + } + "option_history_ohlc" => "Fetch intraday OHLC bars for an option contract.".into(), + "option_history_trade" => "Fetch all trades for an option contract on a given date.".into(), + "option_history_quote" => { + "Fetch NBBO quotes for an option contract on a given date.".into() + } + "option_history_trade_quote" => { + "Fetch combined trade + quote ticks for an option contract.".into() + } + "option_history_open_interest" => { + "Fetch open interest history for an option contract.".into() + } + // Option History Greeks + "option_history_greeks_eod" => { + "Fetch end-of-day Greeks history for an option contract.".into() + } + "option_history_greeks_all" => { + "Fetch all Greeks history for an option contract (intraday, sampled by interval)." + .into() + } + "option_history_trade_greeks_all" => { + "Fetch all Greeks on each trade for an option contract.".into() + } + "option_history_greeks_first_order" => { + "Fetch first-order Greeks history (intraday, sampled by interval).".into() + } + "option_history_trade_greeks_first_order" => { + "Fetch first-order Greeks on each trade for an option contract.".into() + } + "option_history_greeks_second_order" => { + "Fetch second-order Greeks history (intraday, sampled by interval).".into() + } + "option_history_trade_greeks_second_order" => { + "Fetch second-order Greeks on each trade for an option contract.".into() + } + "option_history_greeks_third_order" => { + "Fetch third-order Greeks history (intraday, sampled by interval).".into() + } + "option_history_trade_greeks_third_order" => { + "Fetch third-order Greeks on each trade for an option contract.".into() + } + "option_history_greeks_implied_volatility" => { + "Fetch implied volatility history (intraday, sampled by interval).".into() + } + "option_history_trade_greeks_implied_volatility" => { + "Fetch implied volatility on each trade for an option contract.".into() + } + // Option At-Time + "option_at_time_trade" => { + "Fetch the trade at a specific time of day across a date range for an option.".into() + } + "option_at_time_quote" => { + "Fetch the quote at a specific time of day across a date range for an option.".into() + } + // Index + "index_list_symbols" => "List all available index symbols.".into(), + "index_list_dates" => "List available dates for an index symbol.".into(), + "index_snapshot_ohlc" => "Get the latest OHLC snapshot for one or more indices.".into(), + "index_snapshot_price" => "Get the latest price snapshot for one or more indices.".into(), + "index_snapshot_market_value" => { + "Get the latest market value snapshot for one or more indices.".into() + } + "index_history_eod" => "Fetch end-of-day index data for a date range.".into(), + "index_history_ohlc" => "Fetch intraday OHLC bars for an index.".into(), + "index_history_price" => "Fetch intraday price history for an index.".into(), + "index_at_time_price" => { + "Fetch the index price at a specific time of day across a date range.".into() + } + // Calendar + "calendar_open_today" => "Check whether the market is open today.".into(), + "calendar_on_date" => "Get calendar information for a specific date.".into(), + "calendar_year" => "Get calendar information for an entire year.".into(), + // Rate + "interest_rate_history_eod" => "Fetch end-of-day interest rate history.".into(), + // Fallback: auto-generate from method name + _ => { + let words = method.replace('_', " "); + let mut chars = words.chars(); + match chars.next() { + Some(c) => { + format!("{}{}.", c.to_uppercase(), chars.as_str()) + } + None => method.to_string(), + } + } + } +} diff --git a/crates/thetadatadx/build_support/mod.rs b/crates/thetadatadx/build_support/mod.rs new file mode 100644 index 000000000..7cded0dec --- /dev/null +++ b/crates/thetadatadx/build_support/mod.rs @@ -0,0 +1,19 @@ +//! Build-time generator orchestration for `thetadatadx`. +//! +//! The build pipeline has two distinct responsibilities: +//! - generate tick decoders from `endpoint_schema.toml` +//! - generate endpoint-facing surfaces from the explicit endpoint spec plus +//! the upstream wire contract in `proto/external.proto` + +mod endpoints; +mod ticks; + +pub fn run() -> Result<(), Box> { + tonic_prost_build::configure() + .build_server(false) + .compile_protos(&["proto/external.proto"], &["proto"])?; + + endpoints::generate_all()?; + ticks::generate()?; + Ok(()) +} diff --git a/crates/thetadatadx/build_support/ticks.rs b/crates/thetadatadx/build_support/ticks.rs new file mode 100644 index 000000000..92b329335 --- /dev/null +++ b/crates/thetadatadx/build_support/ticks.rs @@ -0,0 +1,379 @@ +// ───────────────────────────────────────────────────────────────────────────── +// Code-generate tick structs + parsers from endpoint_schema.toml +// ───────────────────────────────────────────────────────────────────────────── + +#[derive(Debug, Deserialize)] +struct Schema { + types: HashMap, +} + +#[derive(Debug, Deserialize)] +struct TickTypeDef { + #[serde(rename = "doc")] + _doc: String, + #[serde(rename = "copy")] + _copy: bool, + #[serde(default, rename = "align")] + _align: Option, + parser: String, + #[serde(default)] + required: Vec, + #[serde(default)] + eod_style: bool, + #[serde(default)] + contract_id: bool, + columns: Vec, +} + +#[derive(Debug, Deserialize)] +struct ColumnDef { + name: String, + field: String, + r#type: String, +} + +pub fn generate() -> Result<(), Box> { + let schema_path = "endpoint_schema.toml"; + let schema_str = std::fs::read_to_string(schema_path)?; + let schema: Schema = toml::from_str(&schema_str)?; + + let out_dir = std::env::var("OUT_DIR")?; + + // Collect type names and sort for deterministic output. + let mut type_names: Vec<&String> = schema.types.keys().collect(); + type_names.sort(); + + // ── Generate decode_generated.rs (parser functions) ────────────────────── + // Struct definitions now live in the `tdbe` crate (crates/tdbe/src/types/tick.rs). + // Only parser functions are generated here. + let mut parsers = String::new(); + parsers.push_str( + "// Auto-generated by build.rs from endpoint_schema.toml \u{2014} do not edit manually.\n\n", + ); + // Note: tick types are imported by decode.rs via `use tdbe::types::tick::*;` + // No additional imports needed in the generated code. + + for type_name in &type_names { + let def = &schema.types[*type_name]; + generate_parser(&mut parsers, type_name, def); + } + + let parsers_dest = Path::new(&out_dir).join("decode_generated.rs"); + std::fs::write(&parsers_dest, &parsers)?; + + // Rerun if schema changes + println!("cargo:rerun-if-changed={schema_path}"); + + Ok(()) +} + +/// Generate a single parser function. +/// +/// Price columns are decoded to `f64` during extraction using the wire +/// `price_type` (internally fetched, never exposed on the public struct). +// Reason: code generator -- the match dispatch over column types cannot be meaningfully split. +#[allow(clippy::too_many_lines)] +fn generate_parser(out: &mut String, type_name: &str, def: &TickTypeDef) { + let fn_name = &def.parser; + + writeln!( + out, + "pub fn {fn_name}(table: &crate::proto::DataTable) -> Vec<{type_name}> {{" + ) + .unwrap(); + + // If eod_style, emit the local eod_num and eod_price helpers inline. + if def.eod_style { + out.push_str(" // EOD rows may have Price-typed cells or plain Number cells.\n"); + out.push_str(" fn eod_num(row: &crate::proto::DataValueList, idx: usize) -> i32 {\n"); + out.push_str(" row.values\n"); + out.push_str(" .get(idx)\n"); + out.push_str(" .and_then(|dv| dv.data_type.as_ref())\n"); + out.push_str(" .and_then(|dt| match dt {\n"); + out.push_str( + " crate::proto::data_value::DataType::Number(n) => Some(*n as i32),\n", + ); + out.push_str( + " crate::proto::data_value::DataType::Price(p) => Some(p.value),\n", + ); + out.push_str( + " crate::proto::data_value::DataType::Timestamp(ts) => Some(crate::decode::timestamp_to_date(ts.epoch_ms)),\n", + ); + out.push_str(" _ => None,\n"); + out.push_str(" })\n"); + out.push_str(" .unwrap_or(0)\n"); + out.push_str(" }\n\n"); + + out.push_str(" // EOD price field: decode to f64 from Price or Number cell.\n"); + out.push_str(" fn eod_price(row: &crate::proto::DataValueList, idx: usize) -> f64 {\n"); + out.push_str(" row.values\n"); + out.push_str(" .get(idx)\n"); + out.push_str(" .and_then(|dv| dv.data_type.as_ref())\n"); + out.push_str(" .map(|dt| match dt {\n"); + out.push_str(" crate::proto::data_value::DataType::Price(p) => {\n"); + out.push_str( + " tdbe::types::price::Price::new(p.value, p.r#type).to_f64()\n", + ); + out.push_str(" }\n"); + out.push_str( + " crate::proto::data_value::DataType::Number(n) => *n as f64,\n", + ); + out.push_str(" _ => 0.0,\n"); + out.push_str(" })\n"); + out.push_str(" .unwrap_or(0.0)\n"); + out.push_str(" }\n\n"); + } + + out.push_str(" let h: Vec<&str> = table.headers.iter().map(|s| s.as_str()).collect();\n"); + + // For eod_style, use the shared find_header() from decode.rs. + if def.eod_style { + out.push_str(" let find = |name: &str| find_header(&h, name);\n\n"); + } + + // Determine which columns need helpers. + let needs_opt_number = def + .columns + .iter() + .any(|c| c.r#type == "i32" && !def.required.contains(&c.name) && c.field != "date"); + let needs_opt_float = def + .columns + .iter() + .any(|c| c.r#type == "f64" && !def.required.contains(&c.name)); + let needs_opt_i64 = def + .columns + .iter() + .any(|c| c.r#type == "i64" && !def.required.contains(&c.name)); + + // Emit opt_number / opt_float / opt_i64 helpers (non-eod only). + if !def.eod_style { + if needs_opt_number { + out.push_str(" fn opt_number(row: &crate::proto::DataValueList, idx: Option) -> i32 {\n"); + out.push_str(" match idx {\n"); + out.push_str(" Some(i) => row_number(row, i),\n"); + out.push_str(" None => 0,\n"); + out.push_str(" }\n"); + out.push_str(" }\n\n"); + } + if needs_opt_float { + out.push_str(" fn opt_float(row: &crate::proto::DataValueList, idx: Option) -> f64 {\n"); + out.push_str(" match idx {\n"); + out.push_str(" Some(i) => row_float(row, i),\n"); + out.push_str(" None => 0.0,\n"); + out.push_str(" }\n"); + out.push_str(" }\n\n"); + } + if needs_opt_i64 { + out.push_str( + " fn opt_i64(row: &crate::proto::DataValueList, idx: Option) -> i64 {\n", + ); + out.push_str(" match idx {\n"); + out.push_str(" Some(i) => row_number_i64(row, i),\n"); + out.push_str(" None => 0,\n"); + out.push_str(" }\n"); + out.push_str(" }\n\n"); + } + } + + // Required header guards (non-eod only). + if !def.eod_style { + for req in &def.required { + let var = idx_var_name(req); + writeln!(out, + " let Some({var}) = find_header(&h, \"{req}\") else {{\n return vec![];\n }};" + ).unwrap(); + } + if !def.required.is_empty() { + out.push('\n'); + } + } + + // Declare index variables for all columns. + for col in &def.columns { + let var = idx_var_name(&col.name); + if def.eod_style { + writeln!(out, " let {var} = find(\"{}\");", col.name).unwrap(); + } else if def.required.contains(&col.name) { + // Already declared above as required. + } else { + writeln!(out, " let {var} = find_header(&h, \"{}\");", col.name).unwrap(); + } + } + + // Contract identification columns (wildcard queries inject expiration/strike/right). + if def.contract_id { + out.push_str(" let _cid_exp_idx = h.iter().position(|c| *c == \"expiration\");\n"); + out.push_str(" let _cid_strike_idx = h.iter().position(|c| *c == \"strike\");\n"); + out.push_str(" let _cid_right_idx = h.iter().position(|c| *c == \"right\");\n"); + out.push_str(" let _cid_strike_is_typed = _cid_strike_idx.is_some() && h.contains(&\"strike\");\n"); + } + + // Check if this is QuoteTick (needs midpoint computation). + let is_quote_tick = type_name == "QuoteTick"; + + out.push('\n'); + + // Emit the row mapping. + out.push_str(" table\n"); + out.push_str(" .data_table\n"); + out.push_str(" .iter()\n"); + out.push_str(" .map(|row| {\n"); + + // Struct literal. + if is_quote_tick { + writeln!(out, " let _tick = {type_name} {{").unwrap(); + } else { + writeln!(out, " {type_name} {{").unwrap(); + } + + for col in &def.columns { + let var = idx_var_name(&col.name); + let is_required = def.required.contains(&col.name); + + match col.r#type.as_str() { + "i32" => { + if def.eod_style { + writeln!( + out, + " {}: {var}.map(|i| eod_num(row, i)).unwrap_or(0),", + col.field + ) + .unwrap(); + } else if is_required { + let accessor = if col.field == "date" { + "row_date" + } else { + "row_number" + }; + writeln!( + out, + " {}: {accessor}(row, {var}),", + col.field + ) + .unwrap(); + } else if col.field == "date" { + writeln!( + out, + " {}: {var}.map(|i| row_date(row, i)).unwrap_or(0),", + col.field + ) + .unwrap(); + } else { + writeln!( + out, + " {}: opt_number(row, {var}),", + col.field + ) + .unwrap(); + } + } + "i64" => { + if is_required { + writeln!( + out, + " {}: row_number_i64(row, {var}),", + col.field + ) + .unwrap(); + } else { + writeln!(out, " {}: opt_i64(row, {var}),", col.field).unwrap(); + } + } + "f64" => { + if is_required { + writeln!(out, " {}: row_float(row, {var}),", col.field).unwrap(); + } else { + writeln!(out, " {}: opt_float(row, {var}),", col.field).unwrap(); + } + } + "String" => { + if is_required { + writeln!(out, " {}: row_text(row, {var}),", col.field).unwrap(); + } else { + writeln!( + out, + " {}: {var}.map(|i| row_text(row, i)).unwrap_or_default(),", + col.field + ) + .unwrap(); + } + } + "price" => { + // Decode price to f64 at parse time. + let field = &col.field; + if is_required { + writeln!(out, " {field}: row_price_f64(row, {var}),").unwrap(); + } else { + writeln!(out, " {field}: match {var} {{").unwrap(); + out.push_str(" Some(i) => row_price_f64(row, i),\n"); + out.push_str(" None => 0.0,\n"); + out.push_str(" },\n"); + } + } + "eod_num" => { + writeln!( + out, + " {}: {var}.map(|i| eod_num(row, i)).unwrap_or(0),", + col.field + ) + .unwrap(); + } + "eod_price" => { + // EOD price field: decode to f64 from Price or Number cell. + writeln!( + out, + " {}: {var}.map(|i| eod_price(row, i)).unwrap_or(0.0),", + col.field + ) + .unwrap(); + } + other => panic!("unknown column type '{other}' in parser for {type_name}"), + } + } + + // Contract identification fields (injected when contract_id = true). + if def.contract_id { + out.push_str( + " expiration: _cid_exp_idx.map(|i| { let n = row_number(row, i); if n != 0 { n } else { let s = row_text(row, i); parse_iso_date(&s) } }).unwrap_or(0),\n", + ); + // strike: decode to f64 + out.push_str(" strike: match _cid_strike_idx {\n"); + out.push_str( + " Some(i) if _cid_strike_is_typed => row_price_f64(row, i),\n", + ); + out.push_str(" Some(i) => row_number(row, i) as f64,\n"); + out.push_str(" None => 0.0,\n"); + out.push_str(" },\n"); + out.push_str(" right: _cid_right_idx.map(|i| {\n"); + out.push_str(" let s = row_text(row, i);\n"); + out.push_str( + " if s.is_empty() { 0i32 } else { s.as_bytes()[0] as i32 }\n", + ); + out.push_str(" }).unwrap_or(0),\n"); + } + + if is_quote_tick { + // QuoteTick gets midpoint computed from bid + ask. + out.push_str(" midpoint: 0.0, // placeholder, set below\n"); + out.push_str(" };\n"); + out.push_str(" let mut _tick = _tick;\n"); + out.push_str(" _tick.midpoint = (_tick.bid + _tick.ask) / 2.0;\n"); + out.push_str(" _tick\n"); + } else { + out.push_str(" }\n"); + } + + out.push_str(" })\n"); + out.push_str(" .collect()\n"); + out.push_str("}\n\n"); +} + +/// Convert a column name to a consistent index variable name. +fn idx_var_name(name: &str) -> String { + format!("{name}_idx") +} +use std::collections::HashMap; +use std::fmt::Write as _; +use std::path::Path; + +use serde::Deserialize; diff --git a/crates/thetadatadx/endpoint_surface.toml b/crates/thetadatadx/endpoint_surface.toml new file mode 100644 index 000000000..a0cca7940 --- /dev/null +++ b/crates/thetadatadx/endpoint_surface.toml @@ -0,0 +1,1372 @@ +# Explicit endpoint surface specification for ThetaDataDx. +# +# This file is the normalized source of truth for endpoint metadata, +# parameter semantics, REST paths, projection call-shapes, and reusable +# endpoint families. +# The build pipeline validates it against `proto/external.proto` and +# combines it with wire metadata extracted from the proto. + +version = 2 + +# Reusable parameter groups. + +[param_groups.request_type] + +[[param_groups.request_type.params]] +name = "request_type" +description = "Request type: EOD, TRADE, QUOTE, OHLC, etc." +param_type = "RequestType" +required = true +binding = "method" + +[param_groups.stock_symbol] + +[[param_groups.stock_symbol.params]] +name = "symbol" +description = "Ticker symbol (e.g. AAPL)" +param_type = "Symbol" +required = true +binding = "method" + +[param_groups.multi_symbol] + +[[param_groups.multi_symbol.params]] +name = "symbol" +description = "Comma-separated ticker symbols (e.g. AAPL,MSFT)" +param_type = "Symbols" +required = true +binding = "method" +arg_name = "symbols" + +[param_groups.expiration_param] + +[[param_groups.expiration_param.params]] +name = "expiration" +description = "Expiration date YYYYMMDD" +param_type = "Expiration" +required = true +binding = "method" + +[param_groups.strike_param] + +[[param_groups.strike_param.params]] +name = "strike" +description = "Strike price (raw integer)" +param_type = "Strike" +required = true +binding = "method" + +[param_groups.right_param] + +[[param_groups.right_param.params]] +name = "right" +description = "C for call, P for put" +param_type = "Right" +required = true +binding = "method" + +[param_groups.contract_identity] + +[[param_groups.contract_identity.params]] +use = "stock_symbol" + +[[param_groups.contract_identity.params]] +use = "expiration_param" + +[[param_groups.contract_identity.params]] +use = "strike_param" + +[[param_groups.contract_identity.params]] +use = "right_param" + +[param_groups.required_date] + +[[param_groups.required_date.params]] +name = "date" +description = "Date YYYYMMDD" +param_type = "Date" +required = true +binding = "method" + +[param_groups.interval] + +[[param_groups.interval.params]] +name = "interval" +description = "Accepts milliseconds (60000) or shorthand (1m). Presets: 100ms, 500ms, 1s, 5s, 10s, 15s, 30s, 1m, 5m, 10m, 15m, 30m, 1h." +param_type = "Interval" +required = true +binding = "method" + +[param_groups.start_time_filter] + +[[param_groups.start_time_filter.params]] +name = "start_time" +description = "Start time filter" +param_type = "Str" +required = false +binding = "builder" +default = "09:30:00" + +[param_groups.end_time_filter] + +[[param_groups.end_time_filter.params]] +name = "end_time" +description = "End time filter" +param_type = "Str" +required = false +binding = "builder" +default = "16:00:00" + +[param_groups.history_time_filters] + +[[param_groups.history_time_filters.params]] +use = "start_time_filter" + +[[param_groups.history_time_filters.params]] +use = "end_time_filter" + +[param_groups.stock_venue_filter] + +[[param_groups.stock_venue_filter.params]] +name = "venue" +description = "Venue/exchange filter" +param_type = "Str" +required = false +binding = "builder" + +[param_groups.min_time_filter] + +[[param_groups.min_time_filter.params]] +name = "min_time" +description = "Minimum time filter" +param_type = "Str" +required = false +binding = "builder" + +[param_groups.start_date_required_alias] + +[[param_groups.start_date_required_alias.params]] +name = "start_date" +description = "Start date YYYYMMDD" +param_type = "Date" +required = true +binding = "method" +arg_name = "start" + +[param_groups.end_date_required_alias] + +[[param_groups.end_date_required_alias.params]] +name = "end_date" +description = "End date YYYYMMDD" +param_type = "Date" +required = true +binding = "method" +arg_name = "end" + +[param_groups.date_range_required_aliases] + +[[param_groups.date_range_required_aliases.params]] +use = "start_date_required_alias" + +[[param_groups.date_range_required_aliases.params]] +use = "end_date_required_alias" + +[param_groups.start_date_required] + +[[param_groups.start_date_required.params]] +name = "start_date" +description = "Start date YYYYMMDD" +param_type = "Date" +required = true +binding = "method" + +[param_groups.end_date_required] + +[[param_groups.end_date_required.params]] +name = "end_date" +description = "End date YYYYMMDD" +param_type = "Date" +required = true +binding = "method" + +[param_groups.date_range_required] + +[[param_groups.date_range_required.params]] +use = "start_date_required" + +[[param_groups.date_range_required.params]] +use = "end_date_required" + +[param_groups.start_date_optional] + +[[param_groups.start_date_optional.params]] +name = "start_date" +description = "Start date YYYYMMDD" +param_type = "Date" +required = false +binding = "builder" + +[param_groups.end_date_optional] + +[[param_groups.end_date_optional.params]] +name = "end_date" +description = "End date YYYYMMDD" +param_type = "Date" +required = false +binding = "builder" + +[param_groups.date_range_optional] + +[[param_groups.date_range_optional.params]] +use = "start_date_optional" + +[[param_groups.date_range_optional.params]] +use = "end_date_optional" + +[param_groups.time_of_day] + +[[param_groups.time_of_day.params]] +name = "time_of_day" +description = "Milliseconds from midnight ET (e.g. 34200000 = 9:30 AM)" +param_type = "Str" +required = true +binding = "method" + +[param_groups.exclusive_flag] + +[[param_groups.exclusive_flag.params]] +name = "exclusive" +description = "Exclusive time boundary" +param_type = "Bool" +required = false +binding = "builder" + +[param_groups.max_dte_filter] + +[[param_groups.max_dte_filter.params]] +name = "max_dte" +description = "Maximum days to expiration" +param_type = "Int" +required = false +binding = "builder" + +[param_groups.strike_range_filter] + +[[param_groups.strike_range_filter.params]] +name = "strike_range" +description = "Strike range filter" +param_type = "Int" +required = false +binding = "builder" + +[param_groups.annual_dividend_input] + +[[param_groups.annual_dividend_input.params]] +name = "annual_dividend" +description = "Annual dividend" +param_type = "Float" +required = false +binding = "builder" + +[param_groups.rate_type_input] + +[[param_groups.rate_type_input.params]] +name = "rate_type" +description = "Rate type" +param_type = "Str" +required = false +binding = "builder" + +[param_groups.rate_value_input] + +[[param_groups.rate_value_input.params]] +name = "rate_value" +description = "Rate value" +param_type = "Float" +required = false +binding = "builder" + +[param_groups.option_greeks_rate_inputs] + +[[param_groups.option_greeks_rate_inputs.params]] +use = "annual_dividend_input" + +[[param_groups.option_greeks_rate_inputs.params]] +use = "rate_type_input" + +[[param_groups.option_greeks_rate_inputs.params]] +use = "rate_value_input" + +[param_groups.stock_price_input] + +[[param_groups.stock_price_input.params]] +name = "stock_price" +description = "Stock price" +param_type = "Float" +required = false +binding = "builder" + +[param_groups.version_input] + +[[param_groups.version_input.params]] +name = "version" +description = "Greeks model version" +param_type = "Str" +required = false +binding = "builder" + +[param_groups.option_snapshot_greeks_inputs] + +[[param_groups.option_snapshot_greeks_inputs.params]] +use = "option_greeks_rate_inputs" + +[[param_groups.option_snapshot_greeks_inputs.params]] +use = "stock_price_input" + +[[param_groups.option_snapshot_greeks_inputs.params]] +use = "version_input" + +[param_groups.option_history_greeks_inputs] + +[[param_groups.option_history_greeks_inputs.params]] +use = "option_greeks_rate_inputs" + +[[param_groups.option_history_greeks_inputs.params]] +use = "version_input" + +[param_groups.use_market_value_flag] + +[[param_groups.use_market_value_flag.params]] +name = "use_market_value" +description = "Use market value for Greeks" +param_type = "Bool" +required = false +binding = "builder" + +[param_groups.underlyer_use_nbbo_flag] + +[[param_groups.underlyer_use_nbbo_flag.params]] +name = "underlyer_use_nbbo" +description = "Use NBBO for underlyer price" +param_type = "Bool" +required = false +binding = "builder" + +[param_groups.year_param] + +[[param_groups.year_param.params]] +name = "year" +description = "4-digit year (e.g. 2024)" +param_type = "Year" +required = true +binding = "method" + +# Reusable endpoint templates. + +[templates.list_base] +kind = "list" +returns = "StringList" + +[templates.stock_base] +category = "stock" + +[templates.stock_parsed] +extends = "stock_base" +kind = "parsed" + +[templates.stock_list] +extends = "list_base" +category = "stock" +subcategory = "list" + +[templates.stock_snapshot] +extends = "stock_parsed" +subcategory = "snapshot" + +[[templates.stock_snapshot.params]] +use = "multi_symbol" + +[[templates.stock_snapshot.params]] +use = "stock_venue_filter" + +[[templates.stock_snapshot.params]] +use = "min_time_filter" + +[templates.stock_history_eod] +extends = "stock_parsed" +subcategory = "history" + +[[templates.stock_history_eod.params]] +use = "stock_symbol" + +[[templates.stock_history_eod.params]] +use = "date_range_required_aliases" + +[templates.stock_history_intraday_interval] +extends = "stock_parsed" +subcategory = "history" + +[[templates.stock_history_intraday_interval.params]] +use = "stock_symbol" + +[[templates.stock_history_intraday_interval.params]] +use = "required_date" + +[[templates.stock_history_intraday_interval.params]] +use = "interval" + +[[templates.stock_history_intraday_interval.params]] +use = "history_time_filters" + +[[templates.stock_history_intraday_interval.params]] +use = "stock_venue_filter" + +[[templates.stock_history_intraday_interval.params]] +use = "date_range_optional" + +[templates.stock_history_intraday_ticks] +extends = "stock_parsed" +subcategory = "history" + +[[templates.stock_history_intraday_ticks.params]] +use = "stock_symbol" + +[[templates.stock_history_intraday_ticks.params]] +use = "required_date" + +[[templates.stock_history_intraday_ticks.params]] +use = "history_time_filters" + +[[templates.stock_history_intraday_ticks.params]] +use = "stock_venue_filter" + +[[templates.stock_history_intraday_ticks.params]] +use = "date_range_optional" + +[templates.stock_history_trade_quote] +extends = "stock_parsed" +subcategory = "history" + +[[templates.stock_history_trade_quote.params]] +use = "stock_symbol" + +[[templates.stock_history_trade_quote.params]] +use = "required_date" + +[[templates.stock_history_trade_quote.params]] +use = "history_time_filters" + +[[templates.stock_history_trade_quote.params]] +use = "exclusive_flag" + +[[templates.stock_history_trade_quote.params]] +use = "stock_venue_filter" + +[[templates.stock_history_trade_quote.params]] +use = "date_range_optional" + +[templates.stock_at_time] +extends = "stock_parsed" +subcategory = "at_time" + +[[templates.stock_at_time.params]] +use = "stock_symbol" + +[[templates.stock_at_time.params]] +use = "date_range_required" + +[[templates.stock_at_time.params]] +use = "time_of_day" + +[[templates.stock_at_time.params]] +use = "stock_venue_filter" + +[templates.stock_history_ohlc_range] +extends = "stock_parsed" +subcategory = "history" + +[[templates.stock_history_ohlc_range.params]] +use = "stock_symbol" + +[[templates.stock_history_ohlc_range.params]] +use = "date_range_required" + +[[templates.stock_history_ohlc_range.params]] +use = "interval" + +[[templates.stock_history_ohlc_range.params]] +use = "history_time_filters" + +[[templates.stock_history_ohlc_range.params]] +use = "stock_venue_filter" + +[templates.option_base] +category = "option" + +[templates.option_parsed] +extends = "option_base" +kind = "parsed" + +[templates.option_list] +extends = "list_base" +category = "option" +subcategory = "list" + +[templates.option_list_parsed] +extends = "option_parsed" +subcategory = "list" + +[templates.option_snapshot_standard] +extends = "option_parsed" +subcategory = "snapshot" + +[[templates.option_snapshot_standard.params]] +use = "contract_identity" + +[[templates.option_snapshot_standard.params]] +use = "max_dte_filter" + +[[templates.option_snapshot_standard.params]] +use = "strike_range_filter" + +[[templates.option_snapshot_standard.params]] +use = "min_time_filter" + +[templates.option_snapshot_trade] +extends = "option_parsed" +subcategory = "snapshot" + +[[templates.option_snapshot_trade.params]] +use = "contract_identity" + +[[templates.option_snapshot_trade.params]] +use = "strike_range_filter" + +[[templates.option_snapshot_trade.params]] +use = "min_time_filter" + +[templates.option_snapshot_greeks] +extends = "option_parsed" +subcategory = "snapshot_greeks" + +[[templates.option_snapshot_greeks.params]] +use = "contract_identity" + +[[templates.option_snapshot_greeks.params]] +use = "option_snapshot_greeks_inputs" + +[[templates.option_snapshot_greeks.params]] +use = "max_dte_filter" + +[[templates.option_snapshot_greeks.params]] +use = "strike_range_filter" + +[[templates.option_snapshot_greeks.params]] +use = "min_time_filter" + +[[templates.option_snapshot_greeks.params]] +use = "use_market_value_flag" + +[templates.option_history_eod] +extends = "option_parsed" +subcategory = "history" + +[[templates.option_history_eod.params]] +use = "contract_identity" + +[[templates.option_history_eod.params]] +use = "date_range_required_aliases" + +[[templates.option_history_eod.params]] +use = "max_dte_filter" + +[[templates.option_history_eod.params]] +use = "strike_range_filter" + +[templates.option_history_ohlc] +extends = "option_parsed" +subcategory = "history" + +[[templates.option_history_ohlc.params]] +use = "contract_identity" + +[[templates.option_history_ohlc.params]] +use = "required_date" + +[[templates.option_history_ohlc.params]] +use = "interval" + +[[templates.option_history_ohlc.params]] +use = "history_time_filters" + +[[templates.option_history_ohlc.params]] +use = "strike_range_filter" + +[[templates.option_history_ohlc.params]] +use = "date_range_optional" + +[templates.option_history_trade] +extends = "option_parsed" +subcategory = "history" + +[[templates.option_history_trade.params]] +use = "contract_identity" + +[[templates.option_history_trade.params]] +use = "required_date" + +[[templates.option_history_trade.params]] +use = "history_time_filters" + +[[templates.option_history_trade.params]] +use = "max_dte_filter" + +[[templates.option_history_trade.params]] +use = "strike_range_filter" + +[[templates.option_history_trade.params]] +use = "date_range_optional" + +[templates.option_history_quote] +extends = "option_parsed" +subcategory = "history" + +[[templates.option_history_quote.params]] +use = "contract_identity" + +[[templates.option_history_quote.params]] +use = "required_date" + +[[templates.option_history_quote.params]] +use = "interval" + +[[templates.option_history_quote.params]] +use = "history_time_filters" + +[[templates.option_history_quote.params]] +use = "max_dte_filter" + +[[templates.option_history_quote.params]] +use = "strike_range_filter" + +[[templates.option_history_quote.params]] +use = "date_range_optional" + +[templates.option_history_trade_quote] +extends = "option_parsed" +subcategory = "history" + +[[templates.option_history_trade_quote.params]] +use = "contract_identity" + +[[templates.option_history_trade_quote.params]] +use = "required_date" + +[[templates.option_history_trade_quote.params]] +use = "history_time_filters" + +[[templates.option_history_trade_quote.params]] +use = "exclusive_flag" + +[[templates.option_history_trade_quote.params]] +use = "max_dte_filter" + +[[templates.option_history_trade_quote.params]] +use = "strike_range_filter" + +[[templates.option_history_trade_quote.params]] +use = "date_range_optional" + +[templates.option_history_open_interest] +extends = "option_parsed" +subcategory = "history" + +[[templates.option_history_open_interest.params]] +use = "contract_identity" + +[[templates.option_history_open_interest.params]] +use = "required_date" + +[[templates.option_history_open_interest.params]] +use = "max_dte_filter" + +[[templates.option_history_open_interest.params]] +use = "strike_range_filter" + +[[templates.option_history_open_interest.params]] +use = "date_range_optional" + +[templates.option_history_greeks_eod] +extends = "option_parsed" +subcategory = "history_greeks" + +[[templates.option_history_greeks_eod.params]] +use = "contract_identity" + +[[templates.option_history_greeks_eod.params]] +use = "date_range_required" + +[[templates.option_history_greeks_eod.params]] +use = "option_history_greeks_inputs" + +[[templates.option_history_greeks_eod.params]] +use = "underlyer_use_nbbo_flag" + +[[templates.option_history_greeks_eod.params]] +use = "max_dte_filter" + +[[templates.option_history_greeks_eod.params]] +use = "strike_range_filter" + +[templates.option_history_greeks] +extends = "option_parsed" +subcategory = "history_greeks" + +[[templates.option_history_greeks.params]] +use = "contract_identity" + +[[templates.option_history_greeks.params]] +use = "required_date" + +[[templates.option_history_greeks.params]] +use = "interval" + +[[templates.option_history_greeks.params]] +use = "history_time_filters" + +[[templates.option_history_greeks.params]] +use = "option_history_greeks_inputs" + +[[templates.option_history_greeks.params]] +use = "strike_range_filter" + +[[templates.option_history_greeks.params]] +use = "date_range_optional" + +[templates.option_history_trade_greeks] +extends = "option_parsed" +subcategory = "history_trade_greeks" + +[[templates.option_history_trade_greeks.params]] +use = "contract_identity" + +[[templates.option_history_trade_greeks.params]] +use = "required_date" + +[[templates.option_history_trade_greeks.params]] +use = "history_time_filters" + +[[templates.option_history_trade_greeks.params]] +use = "option_history_greeks_inputs" + +[[templates.option_history_trade_greeks.params]] +use = "max_dte_filter" + +[[templates.option_history_trade_greeks.params]] +use = "strike_range_filter" + +[[templates.option_history_trade_greeks.params]] +use = "date_range_optional" + +[templates.option_at_time] +extends = "option_parsed" +subcategory = "at_time" + +[[templates.option_at_time.params]] +use = "contract_identity" + +[[templates.option_at_time.params]] +use = "date_range_required" + +[[templates.option_at_time.params]] +use = "time_of_day" + +[[templates.option_at_time.params]] +use = "max_dte_filter" + +[[templates.option_at_time.params]] +use = "strike_range_filter" + +[templates.index_base] +category = "index" + +[templates.index_parsed] +extends = "index_base" +kind = "parsed" + +[templates.index_list] +extends = "list_base" +category = "index" +subcategory = "list" + +[templates.index_snapshot] +extends = "index_parsed" +subcategory = "snapshot" + +[[templates.index_snapshot.params]] +use = "multi_symbol" + +[[templates.index_snapshot.params]] +use = "min_time_filter" + +[templates.index_history_eod] +extends = "index_parsed" +subcategory = "history" + +[[templates.index_history_eod.params]] +use = "stock_symbol" + +[[templates.index_history_eod.params]] +use = "date_range_required_aliases" + +[templates.index_history_ohlc] +extends = "index_parsed" +subcategory = "history" + +[[templates.index_history_ohlc.params]] +use = "stock_symbol" + +[[templates.index_history_ohlc.params]] +use = "date_range_required" + +[[templates.index_history_ohlc.params]] +use = "interval" + +[[templates.index_history_ohlc.params]] +use = "history_time_filters" + +[templates.index_history_price] +extends = "index_parsed" +subcategory = "history" + +[[templates.index_history_price.params]] +use = "required_date" + +[[templates.index_history_price.params]] +use = "stock_symbol" + +[[templates.index_history_price.params]] +use = "interval" + +[[templates.index_history_price.params]] +use = "history_time_filters" + +[[templates.index_history_price.params]] +use = "date_range_optional" + +[templates.index_at_time] +extends = "index_parsed" +subcategory = "at_time" + +[[templates.index_at_time.params]] +use = "stock_symbol" + +[[templates.index_at_time.params]] +use = "date_range_required" + +[[templates.index_at_time.params]] +use = "time_of_day" + +[templates.calendar_base] +category = "calendar" + +[templates.calendar_parsed] +extends = "calendar_base" +kind = "parsed" + +[templates.calendar_status] +extends = "calendar_parsed" +subcategory = "status" + +[templates.calendar_query] +extends = "calendar_parsed" +subcategory = "query" + +[templates.rate_base] +category = "rate" + +[templates.rate_parsed] +extends = "rate_base" +kind = "parsed" + +[templates.rate_history_eod] +extends = "rate_parsed" +subcategory = "history" + +[[templates.rate_history_eod.params]] +use = "stock_symbol" + +[[templates.rate_history_eod.params]] +use = "date_range_required" + +# Concrete endpoint declarations. + +[[endpoints]] +name = "stock_list_symbols" +template = "stock_list" +description = "List all available stock ticker symbols." +rest_path = "/v3/stock/list/symbols" +list_column = "symbol" + +[[endpoints]] +name = "stock_list_dates" +template = "stock_list" +description = "List available dates for a stock by request type (EOD, TRADE, QUOTE, etc.)." +rest_path = "/v3/stock/list/dates" +list_column = "date" + +[[endpoints.params]] +use = "request_type" + +[[endpoints.params]] +use = "stock_symbol" + +[[endpoints]] +name = "stock_snapshot_ohlc" +template = "stock_snapshot" +description = "Get the latest OHLC snapshot for one or more stocks." +rest_path = "/v3/stock/snapshot/ohlc" +returns = "OhlcTicks" + +[[endpoints]] +name = "stock_snapshot_trade" +template = "stock_snapshot" +description = "Get the latest trade snapshot for one or more stocks." +rest_path = "/v3/stock/snapshot/trade" +returns = "TradeTicks" + +[[endpoints]] +name = "stock_snapshot_quote" +template = "stock_snapshot" +description = "Get the latest NBBO quote snapshot for one or more stocks." +rest_path = "/v3/stock/snapshot/quote" +returns = "QuoteTicks" + +[[endpoints]] +name = "stock_snapshot_market_value" +template = "stock_snapshot" +description = "Get the latest market value snapshot for one or more stocks." +rest_path = "/v3/stock/snapshot/market_value" +returns = "MarketValueTicks" + +[[endpoints]] +name = "stock_history_eod" +template = "stock_history_eod" +description = "Fetch end-of-day stock data for a date range. Returns OHLCV + bid/ask per trading day." +rest_path = "/v3/stock/history/eod" +returns = "EodTicks" + +[[endpoints]] +name = "stock_history_ohlc" +template = "stock_history_intraday_interval" +description = "Fetch intraday OHLC bars for a stock on a single date." +rest_path = "/v3/stock/history/ohlc" +returns = "OhlcTicks" + +[[endpoints]] +name = "stock_history_trade" +template = "stock_history_intraday_ticks" +description = "Fetch all trades for a stock on a given date." +rest_path = "/v3/stock/history/trade" +returns = "TradeTicks" + +[[endpoints]] +name = "stock_history_quote" +template = "stock_history_intraday_interval" +description = "Fetch NBBO quotes for a stock on a given date at a given interval." +rest_path = "/v3/stock/history/quote" +returns = "QuoteTicks" + +[[endpoints]] +name = "stock_history_trade_quote" +template = "stock_history_trade_quote" +description = "Fetch combined trade + quote ticks for a stock on a given date. Returns raw DataTable." +rest_path = "/v3/stock/history/trade_quote" +returns = "TradeQuoteTicks" + +[[endpoints]] +name = "stock_at_time_trade" +template = "stock_at_time" +description = "Fetch the trade at a specific time of day across a date range." +rest_path = "/v3/stock/at_time/trade" +returns = "TradeTicks" + +[[endpoints]] +name = "stock_at_time_quote" +template = "stock_at_time" +description = "Fetch the quote at a specific time of day across a date range." +rest_path = "/v3/stock/at_time/quote" +returns = "QuoteTicks" + +[[endpoints]] +name = "option_list_symbols" +template = "option_list" +description = "List all available option underlying symbols." +rest_path = "/v3/option/list/symbols" +list_column = "symbol" + +[[endpoints]] +name = "option_list_dates" +template = "option_list" +description = "List available dates for an option contract by request type." +rest_path = "/v3/option/list/dates" +list_column = "date" + +[[endpoints.params]] +use = "request_type" + +[[endpoints.params]] +use = "contract_identity" + +[[endpoints]] +name = "option_list_expirations" +template = "option_list" +description = "List available expiration dates for an option underlying." +rest_path = "/v3/option/list/expirations" +list_column = "expiration" + +[[endpoints.params]] +use = "stock_symbol" + +[[endpoints]] +name = "option_list_strikes" +template = "option_list" +description = "List available strike prices for an option at a given expiration." +rest_path = "/v3/option/list/strikes" +list_column = "strike" + +[[endpoints.params]] +use = "stock_symbol" + +[[endpoints.params]] +use = "expiration_param" + +[[endpoints]] +name = "option_list_contracts" +template = "option_list_parsed" +description = "List all option contracts for a symbol on a given date." +rest_path = "/v3/option/list/contracts" +returns = "OptionContracts" + +[[endpoints.params]] +use = "request_type" + +[[endpoints.params]] +use = "stock_symbol" + +[[endpoints.params]] +use = "required_date" + +[[endpoints.params]] +use = "max_dte_filter" + +[[endpoints]] +name = "option_snapshot_ohlc" +template = "option_snapshot_standard" +description = "Get the latest OHLC snapshot for an option contract." +rest_path = "/v3/option/snapshot/ohlc" +returns = "OhlcTicks" + +[[endpoints]] +name = "option_snapshot_trade" +template = "option_snapshot_trade" +description = "Get the latest trade snapshot for an option contract." +rest_path = "/v3/option/snapshot/trade" +returns = "TradeTicks" + +[[endpoints]] +name = "option_snapshot_quote" +template = "option_snapshot_standard" +description = "Get the latest NBBO quote snapshot for an option contract." +rest_path = "/v3/option/snapshot/quote" +returns = "QuoteTicks" + +[[endpoints]] +name = "option_snapshot_open_interest" +template = "option_snapshot_standard" +description = "Get the latest open interest snapshot for an option contract." +rest_path = "/v3/option/snapshot/open_interest" +returns = "OpenInterestTicks" + +[[endpoints]] +name = "option_snapshot_market_value" +template = "option_snapshot_standard" +description = "Get the latest market value snapshot for an option contract." +rest_path = "/v3/option/snapshot/market_value" +returns = "MarketValueTicks" + +[[endpoints]] +name = "option_snapshot_greeks_implied_volatility" +template = "option_snapshot_greeks" +description = "Get implied volatility snapshot for an option contract (from ThetaData server)." +rest_path = "/v3/option/snapshot/greeks/implied_volatility" +returns = "IvTicks" + +[[endpoints]] +name = "option_snapshot_greeks_all" +template = "option_snapshot_greeks" +description = "Get all Greeks snapshot for an option contract (from ThetaData server)." +rest_path = "/v3/option/snapshot/greeks/all" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_snapshot_greeks_first_order" +template = "option_snapshot_greeks" +description = "Get first-order Greeks snapshot (delta, theta, rho) for an option contract." +rest_path = "/v3/option/snapshot/greeks/first_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_snapshot_greeks_second_order" +template = "option_snapshot_greeks" +description = "Get second-order Greeks snapshot (gamma, vanna, charm) for an option contract." +rest_path = "/v3/option/snapshot/greeks/second_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_snapshot_greeks_third_order" +template = "option_snapshot_greeks" +description = "Get third-order Greeks snapshot (speed, color, ultima) for an option contract." +rest_path = "/v3/option/snapshot/greeks/third_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_eod" +template = "option_history_eod" +description = "Fetch end-of-day option data for a contract over a date range." +rest_path = "/v3/option/history/eod" +returns = "EodTicks" + +[[endpoints]] +name = "option_history_ohlc" +template = "option_history_ohlc" +description = "Fetch intraday OHLC bars for an option contract." +rest_path = "/v3/option/history/ohlc" +returns = "OhlcTicks" + +[[endpoints]] +name = "option_history_trade" +template = "option_history_trade" +description = "Fetch all trades for an option contract on a given date." +rest_path = "/v3/option/history/trade" +returns = "TradeTicks" + +[[endpoints]] +name = "option_history_quote" +template = "option_history_quote" +description = "Fetch NBBO quotes for an option contract on a given date." +rest_path = "/v3/option/history/quote" +returns = "QuoteTicks" + +[[endpoints]] +name = "option_history_trade_quote" +template = "option_history_trade_quote" +description = "Fetch combined trade + quote ticks for an option contract." +rest_path = "/v3/option/history/trade_quote" +returns = "TradeQuoteTicks" + +[[endpoints]] +name = "option_history_open_interest" +template = "option_history_open_interest" +description = "Fetch open interest history for an option contract." +rest_path = "/v3/option/history/open_interest" +returns = "OpenInterestTicks" + +[[endpoints]] +name = "option_history_greeks_eod" +template = "option_history_greeks_eod" +description = "Fetch end-of-day Greeks history for an option contract." +rest_path = "/v3/option/history/greeks/eod" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_greeks_all" +template = "option_history_greeks" +description = "Fetch all Greeks history for an option contract (intraday, sampled by interval)." +rest_path = "/v3/option/history/greeks/all" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_trade_greeks_all" +template = "option_history_trade_greeks" +description = "Fetch all Greeks on each trade for an option contract." +rest_path = "/v3/option/history/trade_greeks/all" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_greeks_first_order" +template = "option_history_greeks" +description = "Fetch first-order Greeks history (intraday, sampled by interval)." +rest_path = "/v3/option/history/greeks/first_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_trade_greeks_first_order" +template = "option_history_trade_greeks" +description = "Fetch first-order Greeks on each trade for an option contract." +rest_path = "/v3/option/history/trade_greeks/first_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_greeks_second_order" +template = "option_history_greeks" +description = "Fetch second-order Greeks history (intraday, sampled by interval)." +rest_path = "/v3/option/history/greeks/second_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_trade_greeks_second_order" +template = "option_history_trade_greeks" +description = "Fetch second-order Greeks on each trade for an option contract." +rest_path = "/v3/option/history/trade_greeks/second_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_greeks_third_order" +template = "option_history_greeks" +description = "Fetch third-order Greeks history (intraday, sampled by interval)." +rest_path = "/v3/option/history/greeks/third_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_trade_greeks_third_order" +template = "option_history_trade_greeks" +description = "Fetch third-order Greeks on each trade for an option contract." +rest_path = "/v3/option/history/trade_greeks/third_order" +returns = "GreeksTicks" + +[[endpoints]] +name = "option_history_greeks_implied_volatility" +template = "option_history_greeks" +description = "Fetch implied volatility history (intraday, sampled by interval)." +rest_path = "/v3/option/history/greeks/implied_volatility" +returns = "IvTicks" + +[[endpoints]] +name = "option_history_trade_greeks_implied_volatility" +template = "option_history_trade_greeks" +description = "Fetch implied volatility on each trade for an option contract." +rest_path = "/v3/option/history/trade_greeks/implied_volatility" +returns = "IvTicks" + +[[endpoints]] +name = "option_at_time_trade" +template = "option_at_time" +description = "Fetch the trade at a specific time of day across a date range for an option." +rest_path = "/v3/option/at_time/trade" +returns = "TradeTicks" + +[[endpoints]] +name = "option_at_time_quote" +template = "option_at_time" +description = "Fetch the quote at a specific time of day across a date range for an option." +rest_path = "/v3/option/at_time/quote" +returns = "QuoteTicks" + +[[endpoints]] +name = "index_list_symbols" +template = "index_list" +description = "List all available index symbols." +rest_path = "/v3/index/list/symbols" +list_column = "symbol" + +[[endpoints]] +name = "index_list_dates" +template = "index_list" +description = "List available dates for an index symbol." +rest_path = "/v3/index/list/dates" +list_column = "date" + +[[endpoints.params]] +use = "stock_symbol" + +[[endpoints]] +name = "index_snapshot_ohlc" +template = "index_snapshot" +description = "Get the latest OHLC snapshot for one or more indices." +rest_path = "/v3/index/snapshot/ohlc" +returns = "OhlcTicks" + +[[endpoints]] +name = "index_snapshot_price" +template = "index_snapshot" +description = "Get the latest price snapshot for one or more indices." +rest_path = "/v3/index/snapshot/price" +returns = "PriceTicks" + +[[endpoints]] +name = "index_snapshot_market_value" +template = "index_snapshot" +description = "Get the latest market value snapshot for one or more indices." +rest_path = "/v3/index/snapshot/market_value" +returns = "MarketValueTicks" + +[[endpoints]] +name = "index_history_eod" +template = "index_history_eod" +description = "Fetch end-of-day index data for a date range." +rest_path = "/v3/index/history/eod" +returns = "EodTicks" + +[[endpoints]] +name = "index_history_ohlc" +template = "index_history_ohlc" +description = "Fetch intraday OHLC bars for an index." +rest_path = "/v3/index/history/ohlc" +returns = "OhlcTicks" + +[[endpoints]] +name = "index_history_price" +template = "index_history_price" +description = "Fetch intraday price history for an index." +rest_path = "/v3/index/history/price" +returns = "PriceTicks" + +[[endpoints]] +name = "index_at_time_price" +template = "index_at_time" +description = "Fetch the index price at a specific time of day across a date range." +rest_path = "/v3/index/at_time/price" +returns = "PriceTicks" + +[[endpoints]] +name = "calendar_open_today" +template = "calendar_status" +description = "Check whether the market is open today." +rest_path = "/v3/calendar/open_today" +returns = "CalendarDays" + +[[endpoints]] +name = "calendar_on_date" +template = "calendar_query" +description = "Get calendar information for a specific date." +rest_path = "/v3/calendar/on_date" +returns = "CalendarDays" + +[[endpoints.params]] +use = "required_date" + +[[endpoints]] +name = "calendar_year" +template = "calendar_query" +description = "Get calendar information for an entire year." +rest_path = "/v3/calendar/year" +returns = "CalendarDays" + +[[endpoints.params]] +use = "year_param" + +[[endpoints]] +name = "interest_rate_history_eod" +template = "rate_history_eod" +description = "Fetch end-of-day interest rate history." +rest_path = "/v3/rate/history/eod" +returns = "InterestRateTicks" + +[[endpoints]] +name = "stock_history_ohlc_range" +template = "stock_history_ohlc_range" +wire_name = "stock_history_ohlc" +description = "Fetch intraday OHLC bars across a date range." +rest_path = "/v3/stock/history/ohlc_range" +returns = "OhlcTicks" diff --git a/crates/thetadatadx/proto/MAINTENANCE.md b/crates/thetadatadx/proto/MAINTENANCE.md index 3d1b2bf39..d0dd84a43 100644 --- a/crates/thetadatadx/proto/MAINTENANCE.md +++ b/crates/thetadatadx/proto/MAINTENANCE.md @@ -6,7 +6,12 @@ type structs, and DataTable parsers across all languages. ## Source of truth -**`external.proto` is the canonical proto file, provided directly by ThetaData engineering.** +**`external.proto` is the canonical wire contract, provided directly by ThetaData engineering.** + +**`endpoint_surface.toml` is the canonical endpoint surface contract inside this repository.** +It owns normalized endpoint names, parameter semantics, REST paths, return kinds, +projection call-shapes, reusable parameter groups, and endpoint templates. The +build validates that surface spec against the wire contract in `external.proto`. We used to maintain reverse-engineered protos (`endpoints.proto` + `v3_endpoints.proto`) extracted from `ThetaTerminalv3.jar` via `FileDescriptor` reflection. Those have been @@ -25,7 +30,9 @@ proto/ MAINTENANCE.md - this file ../endpoint_schema.toml - column schemas for all DataTable-returning endpoints -../build.rs - reads proto/ and endpoint_schema.toml, generates everything +../endpoint_surface.toml - normalized endpoint surface specification +../build.rs - small build entrypoint +../build_support/ - build-time generators and validators ``` ## What happens on `cargo build` @@ -34,15 +41,38 @@ proto/ client stubs and message types. Output: `$OUT_DIR/beta_endpoints.rs`, exposed at `crate::proto`. -2. **Endpoint registry**: `build.rs` parses `external.proto` with regex to extract all - RPC names, parameter types, and return types. Output: `$OUT_DIR/registry_generated.rs`. +2. **Endpoint surface validation + generation**: the build loads + `endpoint_surface.toml`, parses `external.proto` to extract wire metadata, + validates the surface spec against the wire contract, and generates the + endpoint registry, shared endpoint runtime dispatch, and `DirectClient` + endpoint declarations. Outputs: `$OUT_DIR/registry_generated.rs`, + `$OUT_DIR/endpoint_generated.rs`, + `$OUT_DIR/direct_list_endpoints_generated.rs`, + `$OUT_DIR/direct_parsed_endpoints_generated.rs`. -3. **Tick type codegen**: `build.rs` reads `endpoint_schema.toml` and generates typed Rust - structs and DataTable parser functions. Output: `$OUT_DIR/tick_generated.rs`, - `$OUT_DIR/decode_generated.rs`. +3. **Tick parser codegen**: the build reads `endpoint_schema.toml` and generates + `DataTable` parser functions. Output: `$OUT_DIR/decode_generated.rs`. + The public tick structs live in `crates/tdbe/src/types/tick.rs` and must stay + aligned with that schema. All three steps are automatic. Just run `cargo build`. +## Endpoint surface spec structure + +`endpoint_surface.toml` is intentionally more expressive than the upstream +proto. It supports three layers: + +1. **`param_groups.*`** for reusable parameter blocks such as contract + identity, date ranges, or common builder filters. +2. **`templates.*`** for reusable endpoint families such as stock snapshots or + option Greeks history. Templates may inherit from each other with `extends`. +3. **`[[endpoints]]`** for concrete endpoint declarations that bind a name, + description, rest path, return kind, and any endpoint-specific overrides. + +The generator expands groups and templates first, then validates the fully +resolved endpoint against `external.proto`. Cycles, unknown references, unused +groups/templates, and invalid overrides fail the build. + ## How to: add a new column to an existing endpoint Example: ThetaData adds a `vwap` column to the EOD response. @@ -101,25 +131,16 @@ columns = [ If the response reuses an existing layout (e.g., OHLC bars), skip this step and use the existing type. -**Step 3 — Wire it up** - -In `src/direct.rs`, add: -```rust -parsed_endpoint! { - /// Fetch VWAP history for a stock. - fn stock_history_vwap(symbol: str, start: str, end: str, interval: str) -> Vec; - grpc: get_stock_history_vwap; - request: StockHistoryVwapRequest; - query: StockHistoryVwapRequestQuery { - symbol: symbol.to_string(), - start_date: start.to_string(), - end_date: end.to_string(), - interval: interval.to_string(), - }; - parse: decode::parse_vwap_ticks; - dates: start, end; -} -``` +**Step 3 — Add the endpoint surface entry** + +Add a new entry to `../endpoint_surface.toml` describing the normalized endpoint +surface. The build will validate it against the wire contract in +`external.proto` and generate the SDK-facing declarations automatically. + +Prefer reusing existing `param_groups` and `templates` instead of copying whole +parameter blocks. If the new endpoint introduces a new repeated family shape, +add a new template or parameter group first and then reference it from the +concrete endpoint entry. **Step 4 — Build and test** @@ -138,10 +159,9 @@ When ThetaData ships a new version: 1. Back up the current file: `cp external.proto external.proto.bak` 2. Drop in the new `external.proto` 3. Run `cargo build` — if the proto is valid, stubs regenerate automatically -4. If any RPCs were renamed or removed, `cargo build` will fail with compile errors - pointing to the broken `parsed_endpoint!` / `streaming_endpoint!` / `list_endpoint!` - calls in `direct.rs`. Fix those. -5. If new RPCs were added, add corresponding macro calls (see above). +4. If any RPCs were renamed or removed, `cargo build` will fail validation when + `endpoint_surface.toml` no longer matches the wire contract. Fix the spec. +5. If new RPCs were added, add corresponding entries to `endpoint_surface.toml`. 6. If column schemas changed, update `endpoint_schema.toml` to match. 7. Run `cargo test` to verify everything works. diff --git a/docs/api-reference.md b/docs/api-reference.md index 65e3520ea..b86eb87e8 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -681,7 +681,7 @@ Callers can use either v2-style millisecond strings or v3 shorthand presets inte ### Endpoint Count -ThetaDataDx exposes **61 typed methods** (plus 4 `_stream` variants) covering all 60 gRPC RPCs in `BetaThetaTerminal` plus 1 convenience range-query variant (`stock_history_ohlc_range`). Historical methods are provided via `Deref` (an internal implementation detail) and generated by the `parsed_endpoint!` macro in `direct.rs`. +ThetaDataDx exposes **61 typed methods** (plus 4 `_stream` variants) covering all 60 gRPC RPCs in `BetaThetaTerminal` plus 1 convenience range-query variant (`stock_history_ohlc_range`). Historical methods are provided via `Deref` (an internal implementation detail) and are generated from the checked-in endpoint surface specification validated against the official proto. ### FFI Coverage diff --git a/docs/architecture.md b/docs/architecture.md index c086e72de..b18df406b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -57,7 +57,7 @@ MDDS is a standard gRPC service over TLS, operating on port 443. - **Package**: `BetaEndpoints` - **Service**: `BetaThetaTerminal` -- **Methods**: 60 RPCs, all server-streaming (returning `stream ResponseData`). thetadatadx wraps all 60 gRPC RPCs plus 1 convenience range-query variant = **61 methods** on `ThetaDataDx`, generated via a declarative `parsed_endpoint!` macro (internal implementation uses `DirectClient` via `Deref`). +- **Methods**: 60 RPCs, all server-streaming (returning `stream ResponseData`). thetadatadx wraps all 60 gRPC RPCs plus 1 convenience range-query variant = **61 methods** on `ThetaDataDx`, generated from the checked-in endpoint surface spec (`endpoint_surface.toml`) validated against `external.proto`. The internal `DirectClient` still uses macro-generated builders, but endpoint declarations are no longer hand-maintained. - **Categories**: Stock, Option, Index, Interest Rate, Calendar - each with List, History, Snapshot, AtTime, and Greeks sub-categories ### Request Structure @@ -152,29 +152,47 @@ Three response processing modes are available: - **`for_each_chunk`**: streaming callback that processes each chunk individually without accumulating the full response in memory. - **`_stream` endpoint variants**: `stock_history_trade_stream`, `stock_history_quote_stream`, `option_history_trade_stream`, `option_history_quote_stream` — these combine the gRPC call with `for_each_chunk` processing in a single method call, ideal for endpoints returning millions of rows. -### TOML Codegen Pipeline +### Build-time Generation Pipeline -All 14 tick type structs and their DataTable parsers are generated from `endpoint_schema.toml` at compile time: +ThetaDataDx has two generation pipelines at build time: +- tick parser generation from `endpoint_schema.toml` +- endpoint surface generation from `endpoint_surface.toml` validated against `external.proto` ```mermaid flowchart LR TOML["endpoint_schema.toml
14 tick type definitions
with column schemas"] - BUILD["build.rs"] - STRUCTS["$OUT_DIR/tick_generated.rs
struct definitions"] + SURFACE["endpoint_surface.toml
endpoint spec
groups + templates"] + PROTO["external.proto
official wire contract"] + BUILD["build.rs
delegates to build_support/"] + SUPPORT["build_support/
endpoints.rs + ticks.rs"] PARSERS["$OUT_DIR/decode_generated.rs
parse_* functions"] - TICK["types/tick.rs
include!() + hand-written impl blocks"] + RUNTIME["$OUT_DIR/endpoint_generated.rs
shared endpoint runtime"] + REGISTRY_GEN["$OUT_DIR/registry_generated.rs
EndpointMeta static"] + DIRECT_GEN["$OUT_DIR/direct_*_generated.rs
DirectClient declarations"] + TICK["crates/tdbe/src/types/tick.rs
typed tick structs"] DECODE["decode.rs
include!() + hand-written helpers"] + ENDPOINT["endpoint.rs
include!() + runtime glue"] + REGISTRY["registry.rs
include!() + lookup helpers"] + DIRECT["direct.rs
macro layer + generated declarations"] TOML --> BUILD - BUILD --> STRUCTS - BUILD --> PARSERS - STRUCTS --> TICK + SURFACE --> BUILD + PROTO --> BUILD + BUILD --> SUPPORT + SUPPORT --> PARSERS + SUPPORT --> RUNTIME + SUPPORT --> REGISTRY_GEN + SUPPORT --> DIRECT_GEN PARSERS --> DECODE + TICK --> DECODE + RUNTIME --> ENDPOINT + REGISTRY_GEN --> REGISTRY + DIRECT_GEN --> DIRECT ``` -The 14 generated tick types are: `TradeTick`, `QuoteTick`, `OhlcTick`, `EodTick`, `OpenInterestTick`, `SnapshotTradeTick`, `TradeQuoteTick`, `MarketValueTick`, `GreeksTick`, `IvTick`, `PriceTick`, `CalendarDay`, `InterestRateTick`, `OptionContract`. 10 of these (all except `CalendarDay`, `InterestRateTick`, `PriceTick`, `OptionContract`) carry contract identification fields (`expiration`, `strike`, `right`) populated by the server on wildcard queries. +The 14 tick layouts are: `TradeTick`, `QuoteTick`, `OhlcTick`, `EodTick`, `OpenInterestTick`, `SnapshotTradeTick`, `TradeQuoteTick`, `MarketValueTick`, `GreeksTick`, `IvTick`, `PriceTick`, `CalendarDay`, `InterestRateTick`, `OptionContract`. 10 of these (all except `CalendarDay`, `InterestRateTick`, `PriceTick`, `OptionContract`) carry contract identification fields (`expiration`, `strike`, `right`) populated by the server on wildcard queries. -Adding a new tick type requires only adding a TOML table to `endpoint_schema.toml` - no hand-written struct or parser code needed. See `docs/endpoint-schema.md` for the full schema reference. +Adding a new endpoint now means updating the explicit endpoint surface spec rather than hand-wiring matches across multiple transports. See `crates/thetadatadx/proto/MAINTENANCE.md` for the current maintenance flow. ## FPSS Protocol (Real-Time Streaming) @@ -513,30 +531,38 @@ graph TD end UNIFIED["unified.rs
ThetaDataDx — unified entry point
Deref to DirectClient"] - DIRECT["direct.rs
DirectClient (internal) — 61 endpoints
via parsed_endpoint! macro"] + DIRECT["direct.rs
DirectClient (internal) — generated endpoint declarations
on top of builder macros"] + ENDPOINT_RT["endpoint.rs
shared endpoint runtime"] CONFIG["config.rs
DirectConfig"] DECODE["decode.rs
zstd + DataTable parsing
(includes generated parsers)"] REGISTRY["registry.rs
EndpointMeta, ENDPOINTS static"] - subgraph codegen["TOML Codegen"] + subgraph codegen["Build-time Generation"] SCHEMA["endpoint_schema.toml
14 tick type definitions"] - BUILD["build.rs
reads TOML, generates:
tick_generated.rs (structs)
decode_generated.rs (parsers)"] + SURFACE["endpoint_surface.toml
endpoint surface spec
groups + templates"] + BUILD["build.rs
delegates to build_support/"] + SUPPORT["build_support/
endpoints.rs + ticks.rs"] end subgraph proto["proto/"] - P_V1["endpoints.proto
shared types"] - P_V3["v3_endpoints.proto
60 server-streaming RPCs"] + P_EXT["external.proto
official MDDS wire contract
60 server-streaming RPCs"] end end LIB --> TDBE_LIB UNIFIED --> DIRECT + UNIFIED --> ENDPOINT_RT DIRECT --> auth DIRECT --> DECODE DIRECT --> proto SCHEMA --> BUILD - BUILD --> T_TICK - BUILD --> DECODE + SURFACE --> BUILD + P_EXT --> BUILD + BUILD --> SUPPORT + SUPPORT --> DECODE + SUPPORT --> DIRECT + SUPPORT --> REGISTRY + SUPPORT --> ENDPOINT_RT F_MOD --> tdbe_codec F_MOD --> F_CONN F_MOD --> F_FRAME diff --git a/docs/endpoint-schema.md b/docs/endpoint-schema.md index 82fb63823..ed6c5113d 100644 --- a/docs/endpoint-schema.md +++ b/docs/endpoint-schema.md @@ -1,21 +1,24 @@ # Endpoint Schema (`endpoint_schema.toml`) -The file `crates/thetadatadx/endpoint_schema.toml` is the **single source of truth** for all tick type struct definitions and their DataTable parsers. +The file `crates/thetadatadx/endpoint_schema.toml` is the canonical schema for +ThetaData `DataTable` response layouts and the generated parser functions that +decode them into typed ticks. ## What it is A TOML file where each `[types.TypeName]` table describes: -- The Rust struct (fields, types, derives, repr/alignment) +- The expected `DataTable` columns for a tick layout - The parser function that converts a protobuf `DataTable` into a `Vec` -`build.rs` reads this file at compile time and generates two Rust source files into `$OUT_DIR`: -- `tick_generated.rs` - all tick struct definitions +`build.rs` reads this file at compile time and generates one Rust source file into `$OUT_DIR`: - `decode_generated.rs` - all `parse_*` functions -These are included into the crate via `include!()`: -- `src/types/tick.rs` includes `tick_generated.rs` and adds hand-written `impl` blocks +The parser module is included into the crate via `include!()`: - `src/decode.rs` includes `decode_generated.rs` alongside the hand-written helper functions +The public tick structs themselves live in `crates/tdbe/src/types/tick.rs` and +must stay aligned with the schema. + ## Column types | Type | Rust field type | Reader function | Default | @@ -56,9 +59,10 @@ These are included into the crate via `include!()`: 2. Define all columns with their header names, field names, and types 3. Set `parser = "parse_your_new_ticks"` 4. Set `required`, `copy`, `align`, etc. as needed -5. Run `cargo build` - the struct and parser are generated automatically -6. If the tick needs helper methods (like `get_price()`), add an `impl YourNewTick` block in `src/types/tick.rs` after the `include!()` line -7. Wire up the new parser in `src/direct.rs` where needed +5. Run `cargo build` - the parser is generated automatically +6. Add or update the corresponding tick struct in `crates/tdbe/src/types/tick.rs` +7. If the tick needs helper methods, add them in `tdbe` +8. Wire the new layout into `endpoint_surface.toml` / `external.proto` as needed To add a column to an existing type, just add a new entry to that type's `columns` array. @@ -66,18 +70,6 @@ To add a column to an existing type, just add a new entry to that type's `column For each type in the schema: -**Struct** (`tick_generated.rs`): -```rust -/// Doc comment from schema -#[derive(Debug, Clone, Copy)] // Copy if copy = true -#[repr(C, align(64))] // if align is set -pub struct GreeksTick { - pub ms_of_day: i32, - pub implied_volatility: f64, - // ... -} -``` - **Parser** (`decode_generated.rs`): ```rust pub fn parse_greeks_ticks(table: &crate::proto::DataTable) -> Vec { @@ -97,7 +89,7 @@ If ThetaData adds new fields to an existing endpoint's DataTable: If ThetaData adds a completely new endpoint: -1. Add the proto definitions to `proto/v3_endpoints.proto` -2. Add the tick type to `endpoint_schema.toml` -3. Wire up the RPC in `src/direct.rs` -4. Add any needed `impl` blocks in `src/types/tick.rs` +1. Update `proto/external.proto` +2. Add the endpoint entry to `endpoint_surface.toml` +3. Add the tick type to `endpoint_schema.toml` +4. Add or update the corresponding tick struct in `crates/tdbe/src/types/tick.rs` diff --git a/docs/jvm-deviations.md b/docs/jvm-deviations.md index e2f5c0694..a29660ea1 100644 --- a/docs/jvm-deviations.md +++ b/docs/jvm-deviations.md @@ -23,9 +23,9 @@ As of v1.2.0: | | Java | Rust | Impact | |---|---|---|---| -| **Behavior** | Each of the 60 gRPC handlers is hand-coded with per-endpoint request/response logic | All 61 ThetaDataDx methods (60 gRPC RPCs + 1 range-query variant, via `Deref` to internal `DirectClient`) are generated by a declarative `parsed_endpoint!` macro | Wire-identical requests | -| **Source** | Individual handler classes under `net.thetadata.providers` | `direct.rs:parsed_endpoint!` macro | | -| **Rationale** | The Java terminal duplicates boilerplate across 60 handlers (auth injection, QueryInfo setup, response streaming, zstd decompression). The Rust macro eliminates this duplication: each endpoint is a single macro invocation specifying the gRPC method name, request type, and response parser. Adding a new endpoint requires one line instead of ~50. The generated code is wire-identical to hand-written equivalents. | +| **Behavior** | Each of the 60 gRPC handlers is hand-coded with per-endpoint request/response logic | All 61 ThetaDataDx methods (60 gRPC RPCs + 1 range-query variant, via `Deref` to internal `DirectClient`) are generated from `endpoint_surface.toml` plus `external.proto`; `direct.rs` macros remain an internal expansion target | Wire-identical requests | +| **Source** | Individual handler classes under `net.thetadata.providers` | `build_support/endpoints.rs`, `endpoint_surface.toml`, `direct.rs` macro layer | | +| **Rationale** | The Java terminal duplicates boilerplate across 60 handlers (auth injection, QueryInfo setup, response streaming, zstd decompression). ThetaDataDx centralizes the endpoint contract in a checked-in surface spec, validates it against the wire contract, and generates the registry/runtime/client projections from that data. The resulting requests remain wire-identical while making endpoint maintenance auditable and far less repetitive. | ### FPSS Dispatch: Disruptor Ring Buffer diff --git a/docs/macro-guide.md b/docs/macro-guide.md index 2a3b018af..0125e6ce2 100644 --- a/docs/macro-guide.md +++ b/docs/macro-guide.md @@ -1,12 +1,24 @@ # Macro Guide for Contributors -This guide explains the macro system used to generate ThetaDataDx endpoints. -After reading it, you should be able to add a new endpoint from scratch. +This guide explains the macro system used by the generated `DirectClient` +surface. + +> [!IMPORTANT] +> Most contributors should **not** add endpoints by hand with macro invocations. +> The current endpoint workflow is: +> - update `crates/thetadatadx/proto/external.proto` when the wire contract changes +> - update `crates/thetadatadx/endpoint_surface.toml` for the normalized SDK surface +> - update `crates/thetadatadx/endpoint_schema.toml` if a new `DataTable` layout is introduced +> +> The build then generates the registry, shared endpoint runtime, and +> `DirectClient` declarations automatically. This guide is for understanding and +> maintaining the macro layer that those generators target. ## `parsed_endpoint!` -- the core macro -Every non-streaming endpoint is generated by `parsed_endpoint!` in -`crates/thetadatadx/src/direct.rs`. A single invocation generates three things: +Every non-streaming `DirectClient` endpoint ultimately expands through +`parsed_endpoint!` in `crates/thetadatadx/src/direct.rs`. A single invocation +generates three things: 1. **A builder struct** (e.g., `StockHistoryOhlcBuilder`) that holds required params as owned fields and optional params as `Option` fields. @@ -111,53 +123,48 @@ The helper macros (`req_field_type!`, `req_param_type!`, `opt_field_type!`, | `opt_bool` | `Option` | `bool` | Boolean flags | | `string` | `String` | `&str` | Required-with-default string | -## Adding a new endpoint (step by step) +## Adding a new endpoint (current workflow) + +The supported path for new endpoints is spec-driven, not hand-written macro +expansion. ### 1. Define the tick type (if new) Add a `[types.YourTick]` block in `crates/thetadatadx/endpoint_schema.toml`. -`build.rs` generates the struct and `parse_your_ticks()` function automatically. -See `docs/endpoint-schema.md` for the TOML format. The tick structs themselves -live in `crates/tdbe/`. Set `contract_id = true` if the tick type should carry +`build.rs` generates the `parse_your_ticks()` function automatically. See +`docs/endpoint-schema.md` for the TOML format. The tick structs themselves live +in `crates/tdbe/`. Set `contract_id = true` if the tick type should carry contract identification fields (`expiration`/`strike`/`right`). ### 2. Add the protobuf types (if new message) -Edit `crates/thetadatadx/proto/v3_endpoints.proto` to add the request/params +Update `crates/thetadatadx/proto/external.proto` to add the request/params messages. `cargo build` regenerates Rust types. -### 3. Wire up the endpoint +### 3. Add the endpoint surface entry -In `crates/thetadatadx/src/direct.rs`, add: +Update `crates/thetadatadx/endpoint_surface.toml` with the normalized endpoint +name, REST path, return kind, and parameter semantics. Reuse existing +`param_groups` and `templates` where possible instead of copying large parameter +blocks. -```rust -parsed_endpoint! { - /// Fetch historical VWAP data for a stock. - builder StockHistoryVwapBuilder; - fn stock_history_vwap(symbol: str, start: str, end: str) -> Vec; - grpc: get_stock_history_vwap; - request: StockHistoryVwapRequest; - query: StockHistoryVwapParams { - root: symbol.to_string(), - start_date: start.to_string(), - end_date: end.to_string(), - }; - parse: decode::parse_vwap_ticks; - dates: start, end; - optional { - venue: opt_str = None, - } -} -``` +### 4. Build and inspect the generated surfaces + +Run `cargo build`. The generator validates `endpoint_surface.toml` against +`external.proto` and emits the registry, shared endpoint runtime, and +`DirectClient` endpoint declarations. + +You only need to edit the macro layer or `build_support/endpoints.rs` if the +new endpoint cannot be expressed by the existing surface specification model. -### 4. Expose in SDKs +### 5. Expose in SDKs - **FFI**: add `extern "C"` wrapper in `ffi/src/lib.rs` (see FFI macros below) - **Python**: add PyO3 method in `sdks/python/src/lib.rs` - **Go**: add method in `sdks/go/client.go` - **C++**: add in `sdks/cpp/include/thetadx.hpp` and `sdks/cpp/src/thetadx.cpp` -### 5. Update CHANGELOG.md +### 6. Update CHANGELOG.md Add to `[Unreleased]`. diff --git a/docs/reverse-engineering.md b/docs/reverse-engineering.md index d8f80924b..3508b041b 100644 --- a/docs/reverse-engineering.md +++ b/docs/reverse-engineering.md @@ -1,6 +1,15 @@ # Reverse-Engineering Guide -This crate was built by decompiling ThetaData's Java terminal. This document covers how to repeat the process when ThetaData releases a new version. +> [!WARNING] +> This document is **historical**. ThetaDataDx now uses the official +> `crates/thetadatadx/proto/external.proto` provided by ThetaData engineering. +> Current endpoint/proto maintenance should follow +> [`crates/thetadatadx/proto/MAINTENANCE.md`](../crates/thetadatadx/proto/MAINTENANCE.md), +> not the reverse-engineering workflow below. + +This crate was originally bootstrapped by decompiling ThetaData's Java +terminal. This document records that process for provenance and protocol +research; it is no longer the primary update path for the repository. ## Source Terminal Version @@ -98,7 +107,9 @@ java -cp "dump/:classes/" DumpV3Proto > v3_endpoints.proto java -cp "dump/:classes/" DumpV3Proto > endpoints.proto ``` -The extracted `.proto` files go into `proto/` in the crate root. Run `cargo build` to regenerate the Rust bindings via `tonic-prost-build` (configured in `build.rs`). +Historically, the extracted `.proto` files were written into `proto/` in the +crate root. Today, the repository expects the official single-file +`external.proto`; this extraction workflow is kept only as an archival record. ## 4. Key Java Classes and Rust Module Mapping