refactor: drive endpoint generation from explicit surface spec

CONTRIBUTING.md

@@ -19,7 +19,7 @@ Note: `protoc` is required even if you're not modifying `.proto` files, because
 git clone https://github.com/userFRM/ThetaDataDx.git
 cd ThetaDataDx
 
-# Run the full workspace test suite
+# Run the core workspace test suite
 cargo test --workspace
 
 # For integration tests against ThetaData servers, create creds.txt:
@@ -44,16 +44,30 @@ cargo test --workspace
 # 4. FFI build (if modified)
 cargo build --release -p thetadatadx-ffi
 
-# 5. Python SDK (if modified)
-cd sdks/python && maturin develop && cd ../..
+# 5. Extended surfaces (if modified)
+cargo clippy --manifest-path tools/mcp/Cargo.toml -- -D warnings
+cargo test --manifest-path tools/mcp/Cargo.toml
+cargo clippy --manifest-path tools/server/Cargo.toml -- -D warnings
+cargo test --manifest-path tools/server/Cargo.toml
+cargo clippy --manifest-path tools/cli/Cargo.toml -- -D warnings
+cargo test --manifest-path tools/cli/Cargo.toml
+
+# 6. Language SDK smoke checks (if modified)
+cargo check --manifest-path sdks/python/Cargo.toml
+(cd sdks/go && LD_LIBRARY_PATH=../../target/release go test ./...)
+c++ -std=c++17 -fsyntax-only -I sdks/cpp/include sdks/cpp/src/thetadx.cpp
+cmake -S sdks/cpp -B build/cpp
+cmake --build build/cpp --target thetadatadx_cpp
 ```
 
-One-liner for the common case:
+One-liner for the common Rust-only case:
 
 ```bash
 cargo fmt --all -- --check && cargo clippy --workspace -- -D warnings && cargo test --workspace
 ```
 
+For full Linux parity with the current CI `surfaces` job, run the extended-surface and language-SDK checks above as well.
+
 ## Commit Convention
 
 We follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/).
@@ -116,47 +130,44 @@ feat(core)!: replace DirectClient with ThetaDataDx unified client
 
 ## How to Add a New Endpoint
 
-> **Deep dive:** see [`docs/macro-guide.md`](docs/macro-guide.md) for a full
-> explanation of the macro system, type tags, and FFI wrappers.
+> **Deep dive:** see [`docs/macro-guide.md`](docs/macro-guide.md) for the
+> internal macro system and generated builder model.
+
+The endpoint-facing source of truth is now split across:
+- `crates/thetadatadx/proto/external.proto` for the wire contract
+- `crates/thetadatadx/endpoint_surface.toml` for the normalized SDK surface
+- `crates/thetadatadx/endpoint_schema.toml` for DataTable parser layouts
 
-All 61 endpoint methods are generated by the `parsed_endpoint!` macro in `direct.rs`.
-Tick type structs and parsers are generated from `endpoint_schema.toml` by `build.rs`.
+The build expands that metadata into the registry, shared endpoint runtime, and
+`DirectClient` declarations automatically.
 
 1. **Update the proto** (if the endpoint uses a new message type)
-   - Edit `crates/thetadatadx/proto/v3_endpoints.proto`
+   - Update `crates/thetadatadx/proto/external.proto`
    - `cargo build` regenerates Rust types automatically
 
-2. **Add the column schema** (if the response has a new layout)
+2. **Add or update the endpoint surface**
+   - Add an entry to `crates/thetadatadx/endpoint_surface.toml`
+   - Reuse existing `param_groups` / `templates` where possible
+   - `cargo build` validates the declared surface against `external.proto` and generates the registry/runtime/direct surfaces
+
+3. **Add the column schema** (if the response has a new layout)
    - Add a `[types.YourTick]` block to `crates/thetadatadx/endpoint_schema.toml`
-   - `cargo build` generates the struct and parser
+   - `cargo build` generates the parser
    - See `docs/endpoint-schema.md` for the TOML format
    - Note: tick type structs, `Price`, enums, codecs, and Greeks live in `crates/tdbe/`.
      If you add a new tick type or modify existing types, edit `tdbe` first.
 
-3. **Wire up the endpoint in `direct.rs`**
-   ```rust
-   parsed_endpoint! {
-       /// Doc comment.
-       fn stock_history_vwap(symbol: &str, start: &str, end: &str) -> Vec<VwapTick>;
-       grpc: get_stock_history_vwap;
-       request: StockHistoryVwapRequest;
-       query: StockHistoryVwapParams {
-           root: symbol.to_string(),
-           start_date: start.to_string(),
-           end_date: end.to_string(),
-       };
-       parse: decode::parse_vwap_ticks;
-       dates: start, end;
-   }
-   ```
-
-4. **Expose in downstream SDKs**
+4. **Review the generated direct/runtime surfaces**
+   - Most endpoint additions should not require hand-editing `direct.rs`
+   - Only change `build_support/endpoints.rs` or the macro layer if the new endpoint shape cannot be expressed by the existing surface spec
+
+5. **Expose in downstream SDKs**
    - FFI: add `extern "C"` function in `ffi/src/lib.rs`
    - Python: add PyO3 method in `sdks/python/src/lib.rs`
    - Go: add method in `sdks/go/client.go`
    - C++: add method in `sdks/cpp/include/thetadx.hpp` and `sdks/cpp/src/thetadx.cpp`
 
-5. **Update CHANGELOG.md** under `[Unreleased]`
+6. **Update CHANGELOG.md** under `[Unreleased]`
 
 See `crates/thetadatadx/proto/MAINTENANCE.md` for the full step-by-step guide.
 
@@ -173,16 +184,19 @@ Every PR must include:
 - Updated `CHANGELOG.md` if user-facing
 - Updated documentation if any public API changed
 
-## How to Update After a ThetaData Terminal Update
+## How to Update After a ThetaData Protocol Update
 
-When ThetaData releases a new terminal version:
+When ThetaData ships a new official proto revision:
 
-1. Refer to `docs/reverse-engineering.md` for methodology
-2. Drop new proto files in `crates/thetadatadx/proto/`
-3. Update `endpoint_schema.toml` if column schemas changed
-4. Run the full test suite to verify backwards compatibility
+1. Replace `crates/thetadatadx/proto/external.proto`
+2. Update `endpoint_surface.toml` when the normalized SDK surface changes
+3. Update `endpoint_schema.toml` if DataTable column layouts changed
+4. Run the relevant checks from the sections above
 5. See `crates/thetadatadx/proto/MAINTENANCE.md` for the detailed guide
 
+`docs/reverse-engineering.md` is kept as historical context for how the project
+was originally bootstrapped, not as the primary maintenance workflow.
+
 ## Community
 
 Join the ThetaData Discord for questions and discussion: **[discord.thetadata.us](https://discord.thetadata.us/)**

README.md

@@ -29,7 +29,7 @@ No-JVM ThetaData Terminal - native Rust SDK for direct market data access.
 | [`tools/cli/`](tools/cli/) | `tdx` CLI - all 61 endpoints from the command line |
 | [`tools/mcp/`](tools/mcp/) | MCP server - gives LLMs access to 64 tools over JSON-RPC |
 | [`tools/server/`](tools/server/) | REST+WS server - drop-in replacement for the Java terminal |
-| [`docs/`](docs/) | Architecture, API reference, JVM deviations, reverse-engineering guide |
+| [`docs/`](docs/) | Architecture, API reference, JVM deviations, and historical reverse-engineering notes |
 | [`docs-site/`](docs-site/) | mdBook documentation site (deployed to GitHub Pages) |
 | [`notebooks/`](notebooks/) | 7 Jupyter notebooks (101-107) |
 | [`examples/`](examples/) | Example programs and test scripts |
@@ -143,7 +143,7 @@ All endpoints return fully typed native structs in every language. Zero raw JSON
 | [API Reference](docs/api-reference.md) | All 61 methods, 14 tick types, configuration options |
 | [Architecture](docs/architecture.md) | System design, wire protocols, TOML codegen pipeline |
 | [JVM Deviations](docs/jvm-deviations.md) | Intentional differences from the Java terminal |
-| [Reverse-Engineering Guide](docs/reverse-engineering.md) | How to decompile the terminal and extract proto definitions |
+| [Reverse-Engineering Guide](docs/reverse-engineering.md) | Historical archive of the original reverse-engineering process before the official proto handoff |
 | [Endpoint Schema](docs/endpoint-schema.md) | TOML codegen format for adding new types/columns |
 | [Java Class Mapping](docs/java-class-mapping.md) | All 588 Java terminal classes enumerated |
 | [Proto Maintenance](crates/thetadatadx/proto/MAINTENANCE.md) | Guide for ThetaData engineers updating proto files |

crates/thetadatadx/Cargo.toml

@@ -12,7 +12,7 @@ license = "GPL-3.0-or-later"
 keywords = ["thetadata", "market-data", "tick-data", "zero-copy", "low-latency"]
 categories = ["finance", "api-bindings", "network-programming", "encoding"]
 readme = "../../README.md"
-include = ["src/**/*", "proto/**/*", "build.rs", "Cargo.toml", "endpoint_schema.toml"]
+include = ["src/**/*", "proto/**/*", "build.rs", "build_support/**/*", "Cargo.toml", "endpoint_schema.toml", "endpoint_surface.toml"]
 
 [package.metadata.docs.rs]
 all-features = true

crates/thetadatadx/README.md

@@ -31,28 +31,42 @@ tdx.stop_streaming();
 src/
   lib.rs           - public re-exports (ThetaDataDx, Credentials, DirectConfig, Error)
   unified.rs       - ThetaDataDx: single entry point, lazy streaming
-  direct.rs        - DirectClient: 61 gRPC endpoints via parsed_endpoint! macro
+  direct.rs        - DirectClient macros plus generated endpoint declarations
   auth/            - Nexus API authentication, credential parsing
   fpss/            - FPSS streaming client (sync, LMAX Disruptor ring buffer)
   codec/           - FIT nibble encoder/decoder, delta compression
   config.rs        - DirectConfig (server addresses, timeouts, concurrency)
   decode.rs        - DataTable -> typed tick parsing (generated from TOML)
   types/           - Tick structs, Price, enums (generated from TOML)
   greeks.rs        - 22 Black-Scholes Greeks + IV solver
-  registry.rs      - Endpoint metadata (generated from proto at build time)
+  registry.rs      - Endpoint metadata (generated from the endpoint surface spec)
   error.rs         - Error enum
 proto/
-  endpoints.proto      - shared types (DataTable, ResponseData, Price)
-  v3_endpoints.proto   - v3 service (BetaThetaTerminal, 60 RPCs)
-  MAINTENANCE.md       - guide for ThetaData engineers
+  external.proto       - canonical MDDS wire contract from ThetaData
+  MAINTENANCE.md       - endpoint/proto maintenance guide
 endpoint_schema.toml   - single source of truth for tick type definitions
-build.rs               - proto compilation + endpoint registry + TOML codegen
+endpoint_surface.toml  - explicit endpoint surface spec for registry/direct/runtime generation
+build.rs               - small build entrypoint
+build_support/         - build-time generators for tick decoding and endpoint surfaces
 ```
 
 ## TOML Codegen
 
 All 14 tick types and their DataTable parsers are generated at compile time from `endpoint_schema.toml`. Adding a new column is one line in the TOML. See [docs/endpoint-schema.md](../../docs/endpoint-schema.md).
 
+## Endpoint Surface Spec
+
+Endpoint projections are generated from the checked-in `endpoint_surface.toml`
+file, which defines the normalized endpoint surface: names, descriptions,
+parameter semantics, REST paths, return kinds, projection call-shapes, reusable
+parameter groups, and endpoint templates. Templates support inheritance via
+`extends`, so the spec can model repeated endpoint families without copying the
+same parameter blocks across every declaration.
+
+The build pipeline validates that surface spec against `proto/external.proto`
+before generating the registry, shared endpoint runtime, and `DirectClient`
+endpoint declarations.
+
 ## Tick Types
 
 | Type | Fields | Use |