feat(grpc): in-house gRPC transport replacing tonic

[userFRM]

Closes #522.

Summary

In-house gRPC client stack built directly on the h2 crate. The MddsClient and every generated MDDS endpoint now route through this transport; tonic, tonic-prost, and tonic-prost-build are removed from the workspace.

Transport machinery

• Codec<Req, Resp> — phantom-typed wrapper performing prost encode/decode plus the 5-byte gRPC frame header ([1 byte compressed flag][4 bytes BE length][payload]). Rejects any non-zero compressed flag and any oversized frame.
• Status — HTTP/2 trailers parser (grpc-status + grpc-message). Missing or malformed trailers surface as typed errors rather than panics.
• Channel — owns one HTTP/2 connection. connect_tls runs the TLS handshake through tokio-rustls; connect_h2c opens a plaintext connection for sidecars and mock harnesses.
• ServerStreaming — futures_core::Stream adapter over the h2 response body. Reads DATA frames into a BytesMut accumulator, decodes each complete frame through Codec, and on body close awaits trailers to surface either stream-end (OK status) or Err(ChannelError::Rpc) (non-OK status).

Off-reactor decoder pool

  +-------------------+   LMAX ring   +-------------------+
  | h2 receive task   |-------------->| Decoder thread    |
  | (tokio worker)    |  work+reply   | (std::thread,     |
  +-------------------+               |  TLS zstd ctx +   |
                                      |  scratch buffer)  |
                                      +---------+---------+
                                                |
                                                | DataTable
                                                v
                                      +-------------------+
                                      | oneshot::Sender   |
                                      | -> async caller   |
                                      +-------------------+

• DecoderPool runs every response chunk's zstd decompress + DataTable::decode on dedicated std::threads. Each decoder owns one LMAX Disruptor ring + the existing thread-local zstd state in mdds::decode::transport. Producers are MultiProducer clones so concurrent submissions from N tokio tasks stay lock-free in the steady state.
• DecoderWaitStrategy (16 spins + 4 yields + spin-loop hint) tuned for MDDS burst cadence — short enough that submit-to-dequeue handoff is single-digit microseconds, cooperative enough that idle decoders do not burn whole cores between bursts.
• MddsClient::connect wires the pool through ChannelPool::from_channels_with_decoders.
• Sizing knobs on MddsConfig: decoder_threads (0 = auto-size to min(channels, available_parallelism / 2)) and decoder_ring_size (power of two, >= 64, validated via util::ring::check_ring_size).
• util::ring lifted out of fpss::ring so the FPSS read loop and the gRPC decoder pool share one validator.

Hardening

• Per-call deadlines via Channel::server_streaming_with_deadline cover both the open phase (h2 send_request + response head) and the streaming phase (DATA frames + trailers). On elapse the h2 stream is dropped (sending RST_STREAM) and the call surfaces ChannelError::DeadlineExceeded.
• GOAWAY classification: h2::Error::is_go_away / is_remote / is_io surface as ChannelError::ConnectionClosed so consumers can recycle the channel rather than retry on a dead connection.
• Cancellation: dropping a ServerStreaming cancels the underlying h2 stream cleanly. Cancelled decode work is elided — the decoder thread checks oneshot::Sender::is_closed before running the work, so cancelled RPCs do not burn CPU on results no one will read.
• grpc-encoding: identity is the only accepted body encoding; zstd-compressed compressed_data payloads inside ResponseData are decompressed by the decoder pool's dedicated threads.

Pooling

• ChannelPool round-robins over N independent Channels so workloads that exceed the per-connection MAX_CONCURRENT_STREAMS limit fan out across multiple h2 connections. The cursor is an AtomicUsize advanced with Relaxed ordering.
• MddsClient::connect builds a 4-channel pool by default (clamped to the subscription-tier concurrent ceiling). Every endpoint call picks the next channel via MddsClient::channel() — transparent to downstream callers.
• Channels and decoder threads are independent dimensions: 4 channels × 6 decoder threads is the typical production shape on a 12-logical-core host.

Custom codegen replacing tonic-build

• build_support/grpc/ reads proto/mdds.proto, invokes prost-build for the message types, and installs a custom ServiceGenerator that emits one async function per RPC method into $OUT_DIR/beta_endpoints.rs. Each emitted function:

  pub async fn <method_snake_case>(
      channel: &Channel,
      req: <RequestType>,
  ) -> Result<ServerStreaming<<ResponseType>>, ChannelError>
  

  with method path /BetaEndpoints.BetaThetaTerminal/<MethodCamelCase>. The generator rejects unary or client-streaming RPCs at build time so a future proto change cannot silently mis-frame the wire.

• build_support/grpc::check reruns the codegen and confirms the output is byte-identical to the file already on disk. Gated by the THETADATADX_GRPC_CHECK env var so CI can re-run the build with the var set and surface drift as a build failure.

Endpoint migration

• MddsClient::stub() is gone; every generated list_endpoint! / parsed_endpoint! / streaming macro invocation routes through crate::proto::beta_theta_terminal::<method>(client.channel(), request).
• mdds::stream::{collect_stream, for_each_chunk} accept grpc::ServerStreaming<ResponseData> instead of tonic::Streaming<ResponseData>. Each chunk's decode now routes through the stream's attached DecoderHandle.
• error.rs swaps Error::Transport(tonic::transport::Error) for Error::Transport(String) and replaces the From<tonic::Status> impl with From<crate::grpc::Status> + From<crate::grpc::ChannelError>. GrpcStatusKind::from_code(tonic::Code) becomes GrpcStatusKind::from_u32(u32) reading the wire code directly.

Tonic removal

• tonic, tonic-prost, and tonic-prost-build are removed from crates/thetadatadx/Cargo.toml. The inhouse-grpc Cargo feature flag is dropped (the in-house transport is now the only path).
• cargo tree -i tonic returns "no such package"; cargo tree --workspace | grep tonic is empty.

Test plan

• Unit tests for codec: roundtrip, partial header, partial payload, compressed-flag rejection, invalid-flag rejection, oversized-frame rejection, two-frame demultiplex, wire-format parity vs the gRPC spec, plus a proptest over arbitrary Vec<Vec<u8>> payloads.
• Unit tests for status: OK, error-with-message, missing trailer, non-numeric status, non-UTF8 status, non-UTF8 message, Display formatting.
• Unit tests for pool: even distribution, within-one partial cycle, thread-safety under contention, panic on empty.
• Unit tests for decoder pool: empty-pool rejection, invalid ring size rejection, single-response decode, concurrent decode (16 in-flight per decoder), cancelled-request elision, pool clone sharing decoders.
• Integration tests against the mock h2 server: single chunk, multi-chunk in order, non-OK status surfacing as ChannelError::Rpc, connect to closed port, DeadlineExceeded mid-stream, GOAWAY classified as ConnectionClosed, ChannelPool round-robin distribution.
• End-to-end test for stock_list_symbols through the in-house Channel (single chunk and two-chunk merge).
• Error-mapping tests: From<grpc::Status> carries kind + message, unauthenticated kind, deadline routes to Error::Timeout, rpc routes to Error::Grpc.
• cargo fmt --all -- --check
• cargo clippy --workspace --locked -- -D warnings
• cargo clippy --workspace --locked --tests --benches -- -D warnings
• cargo test --workspace --locked
• cargo run -p thetadatadx --features config-file --bin generate_sdk_surfaces --locked -- --check
• python3 scripts/check_docs_consistency.py
• cargo tree -i tonic returns empty

Cross-binding ABI

The Rust transport changes preserve the binding-visible error contract:

• Error::Transport(String) still formats identically — bindings parse the message, not the inner type.
• ChannelError::Rpc { status: Status } keys off status.code(), unchanged.
• New CodecError::Encode only surfaces via ChannelError::Codec(_) which maps to Error::Transport at the FFI boundary.
• Channel::connect_*_with_max_message_size are additive — the original connect_h2c / connect_tls keep their signatures.
• Channel::with_decoder is a builder-style attachment; channels constructed without it fall back to the inline decode path.

Bench numbers

grpc_channel (loopback mock, 100-sample p50)

Endpoint	p50	p95	alloc/call
stock_list_symbols	107.84 µs	108.88 µs	78,127 B
stock_history_eod	58.90 µs	59.68 µs	18,958 B
option_history_quote	71.86 µs	72.70 µs	32,698 B

grpc_concurrent_burst (64-way burst on a 4-channel pool, loopback mock)

Rows	p50 (baseline)	p50 (after)	Δ	Throughput Δ	Alloc/call Δ
256	3.45 ms	2.28 ms	-33.9%	+51.4%	-1.4% (-914 B)
4096	54.86 ms	29.99 ms	-45.3%	+82.9%	-1.2% (-11 KB)
250000	3.612 s	1.820 s	-49.6%	+98.8%	-1.2% (-1.2 MB)

Flamegraph at 4096 rows confirms prost::Message::decode, decompress_response, and ZSTD_decompressBlock_internal frames sit under the decoder thread group (87.3% of total samples) rather than the tokio worker stack (2.5%).

Public surface impact

• thetadatadx::grpc::{Channel, ChannelError, ChannelPool, Codec, CodecError, ServerStreaming, Status, StatusParseError} — new public surface.
• thetadatadx::grpc::{DecoderPool, DecoderHandle, DecoderPoolError, DecoderWaitStrategy, DecodeResult, default_decoder_thread_count} — new public decoder pool surface.
• thetadatadx::grpc::{stock_list_symbols, endpoints::bench_support} — new public functions.
• ChannelPool::from_channels_with_decoders — production constructor attaching a decoder pool to a channel set.
• Channel::with_decoder — builder-style decoder attachment.
• Channel::connect_h2c_with_max_message_size / Channel::connect_tls_with_max_message_size — codec-ceiling-aware constructors; the existing connect_h2c / connect_tls keep their signatures and delegate to the new ones with DEFAULT_MAX_MESSAGE_SIZE.
• MddsConfig::decoder_threads / MddsConfig::decoder_ring_size — new public fields with 0 / 256 production defaults.
• MddsClient::stub is gone (was crate-private; no downstream impact).
• Error::Transport changes from From<tonic::transport::Error> to carrying a String — callers that previously matched the inner error must update.
• inhouse-grpc Cargo feature is removed.
• Cross-binding ABI (Python / TypeScript / C++ / FFI) unchanged.

Allocator and bench knobs

Optional mimalloc-allocator feature

thetadatadx = { features = ["mimalloc-allocator"] } pulls mimalloc into the dependency graph and re-exports MiMalloc at thetadatadx::mimalloc::MiMalloc so the consuming binary can wire it in with one line:

#[global_allocator]
static GLOBAL: thetadatadx::mimalloc::MiMalloc = thetadatadx::mimalloc::MiMalloc;

Default-off — applications that prefer the system allocator pay no compile or link cost. Full opt-in walk-through in docs-site/docs/configuration.md under Performance tuning. Library crates cannot install a #[global_allocator] of their own, so the SDK only provides the re-export.

THETADATADX_BENCH_ROWS env override on grpc_concurrent_burst

The concurrent-burst harness honours THETADATADX_BENCH_ROWS=<n> so the same bench sweeps the small, medium, and large payload regimes without recompiling.