Skip to content

feat(grpc): migrate MDDS endpoints onto in-house transport and remove tonic #525

@userFRM

Description

@userFRM

Follow-up to #522 / #524.

Goal

PR #524 lands the in-house gRPC transport (Channel, Codec, Status,
ServerStreaming), hardening (deadlines, GOAWAY classification,
cancellation), and ChannelPool. This issue tracks the remaining
migration work: get every MDDS endpoint onto the in-house transport
and remove the tonic / tonic-prost / tonic-prost-build deps
entirely.

Scope

Custom codegen replacing tonic-build

  • New build-time step under crates/thetadatadx/build_support/grpc/
    invokes prost-build for message types only, then a custom
    ServiceGenerator emits one async function per RPC method.
  • Output file: \$OUT_DIR/grpc_endpoints_generated.rs, include!'d
    from crates/thetadatadx/src/grpc/endpoints.rs.
  • Each generated function:
    pub async fn <method_snake_case>(
        channel: &Channel,
        req: <RequestType>,
    ) -> Result<ServerStreaming<<ResponseType>>, ChannelError>
  • Idempotent regeneration. Add a build-script --check mode
    (env-var-gated) for CI drift detection.

Migrate every MDDS endpoint

  • mdds/macros.rs list_endpoint! and parsed_endpoint! macros
    switch from self.stub().$grpc(request).await to the generated
    crate::proto::beta_theta_terminal::$grpc_snake(&self.channel, request).await.
  • mdds/client.rs swaps the tonic::transport::Channel field for
    grpc::ChannelPool (MddsClient::connect builds a 4-channel pool
    by default; each call picks the next channel).
  • mdds/stream.rs collect_stream and for_each_chunk switch from
    tonic::Streaming<ResponseData> to
    grpc::ServerStreaming<ResponseData>.
  • Retry / refresh loops in the macros update their error
    classification: ChannelError::DeadlineExceeded
    StatusClass::Transient, ChannelError::ConnectionClosed
    StatusClass::Transient (with channel recycle),
    ChannelError::Rpc { status } → existing from_tonic_status
    logic ported to read grpc::Status::code.

Remove tonic

  • Drop tonic / tonic-prost / tonic-prost-build from
    crates/thetadatadx/Cargo.toml.
  • Remove the From<tonic::Status> impl and the GrpcStatusKind
    mapping in error.rs; replace with From<grpc::Status> reading
    the numeric code directly.
  • After the migration, cargo tree -i tonic returns empty for the
    workspace.

Drop the inhouse-grpc feature flag

  • Once the in-house path is the only path, the feature flag has no
    variants to gate. Optional deps (h2, http, http-body,
    bytes, pin-project-lite, futures-core) move to required.

Update the bench

  • benches/grpc_channel.rs switches to in-house-only and adds 2-3
    representative endpoints beyond stock_list_symbols
    (stock_history_eod, option_history_quote).

Constraints

  • Conventional Commits, factual messages.
  • TDD where the change is testable through the macros' surface
    (deadlines, retry on DeadlineExceeded, retry on
    ConnectionClosed).
  • No #[allow(dead_code)], no TODO / FIXME / XXX in shipped code.
  • cargo fmt --all -- --check, cargo clippy --workspace --locked -- -D warnings, cargo test --workspace --locked,
    cargo run -p thetadatadx --features config-file --bin generate_sdk_surfaces --locked -- --check,
    python3 scripts/check_docs_consistency.py — all green.
  • Cross-binding ABI stays unchanged — Python / TypeScript / C++
    bindings keep working without recompile.

Definition of done

  • All 60+ MDDS endpoints route through grpc::ChannelPool ->
    grpc::Channel::server_streaming.
  • cargo tree -i tonic is empty.
  • The inhouse-grpc Cargo feature is gone (or relabeled if needed
    for downstream consumers).
  • Bench numbers reported in the PR description for at least three
    representative endpoints.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions