diff --git a/A116-virtual-rpcs.md b/A116-virtual-rpcs.md new file mode 100644 index 000000000..1bd295020 --- /dev/null +++ b/A116-virtual-rpcs.md @@ -0,0 +1,397 @@ +gRPC Virtual RPCs on Bidi Streams +--------------------------------- + +* Author(s): Siddharth Nohria +* Approver: a11r +* Status: Draft +* Implemented in: C++ (Prototype) +* Last updated: 2026-03-30 +* Discussion at: + +Table of Contents +----------------- + +* [Abstract](#abstract) +* [Background](#background) +* [Terminology](#terminology) +* [Design](#design) + * [Protobuf API](#protobuf-api) + * [Generic API Support](#generic-api-support) + * [The Session Lifecycle](#the-session-lifecycle) + * [Establishment and Barrier Mechanism](#establishment-and-barrier-mechanism) + * [Deadlines and Cancellations](#deadlines-and-cancellations) + * [Client Retries](#client-retries) + * [Graceful Shutdown](#graceful-shutdown) + * [Data Flow and Multiplexing](#data-flow-and-multiplexing) + * [Wire Format: HTTP/2 on HTTP/2](#wire-format-http2-on-http2) + * [Alternative Wire Format](#alternative-wire-format) + * [Why HTTP/2 on HTTP/2 is Chosen](#why-http2-on-http2-is-chosen) + * [Flow Control](#flow-control) + * [Server-Side Architecture: Dual Server Approach](#server-side-architecture-dual-server-approach) +* [C++ User-Facing APIs](#c-user-facing-apis) + * [Client-Side API](#client-side-api) + * [Server-Side API & Context Propagation](#server-side-api) + * [Context Propagation](#context-propagation) +* [Java User-Facing APIs](#java-user-facing-apis) +* [Go User-Facing APIs](#go-user-facing-apis) + +Abstract +-------- + +gRPC will support multiplexing virtual RPCs over a single common bi-directional +stream. The high-level idea is that by sending the virtual RPCs over the same +stream, we can eliminate significant one-time setup costs associated with each +RPC. In addition, server applications can cache and re-use potentially heavy +per-client application metadata, which are likely to be consistent across all +RPCs from the same client. + +Background +---------- + +Currently, each RPC sent over a gRPC channel incurs per-RPC overheads. +For high-throughput services, operations like authentication, authorization +policy validation, and other repetitive checks consume latency and CPU usage on +a per-RPC basis. When the same clients are sending the RPC methods repeatedly, +these are wasted cycles. Additionally, this allows applications to re-use heavy +per-client application states, which are likely to be consistent across all RPCs +from the same client. + +While applications can implement custom multiplexing over bidirectional streams, +doing so forces owners to reinvent gRPC features, from the core features like +deadlines and cancellations, to the more complex ones like telemetry, flow +control, load shedding etc. Moving this functionality into gRPC as a central +framework avoids duplicated effort and provides a structured mechanism for +high-performance, stateful RPC multiplexing. + +Terminology +----------- + +* **Physical Stream**: The underlying BiDi stream over which all messages are exchanged. +* **Virtual RPC (vRPC) / Stream**: The user-facing single RPC / stream, multiplexed over the physical stream. +* **Client / Server Session**: The internal gRPC state handling the virtual RPCs. +* **Session RPC**: The initial user-facing RPC used to establish and manage the session. +* **Virtual Stub**: Wraps a single physical stream and is used to start virtual RPCs. + +Design +-------- + +### Protobuf API + +We will use a new protobuf syntax: `returns service`. The client will send an +initial Session Request to establish the common context to be used for every +virtual RPC. + +```proto +service FooService { + rpc SessionRequest(ApplicationRequest) returns (service FooVrpcService); +} + +service FooVrpcService { + rpc VrpcUnaryMethod(VrpcRequest) returns (VrpcResponse); + rpc VrpcClientStreamingMethod(stream VrpcRequest) returns (VrpcResponse); + rpc VrpcServerStreamingMethod(VrpcRequest) returns (stream VrpcResponse); + rpc VrpcBidiStreamingMethod(stream VrpcRequest) returns (stream VrpcResponse); +} +``` + +Since the `returns service` syntax will not be available until a future Protobuf +edition, we need to support a workaround way to start sessions. + +### Generic API Support + +The `GenericStub` allows users to send requests without any dependency on +Protobuf, and the corresponding generated code. Clients can directly send +any methods over the stub. This is especially useful for proxies that need to +dynamically route requests without knowing the specific payload types in +advance. + +gRPC will provide a `GenericStub` based API for both the session establishment +request, and the virtual RPCs, allowing clients in all supported languages to +invoke sessions dynamically. + +### The Session Lifecycle + +![Lifecycle Diagram](A116_graphics/vrpc_flow.png) + +#### Establishment and Barrier Mechanism + +The session begins when the client initiates a Session RPC. This RPC establishes +the physical bi-directional stream and allows the client to send session-level +metadata, and an initial payload message. To the client application, this RPC +immediately returns a "Virtual Stub", which the application can then use to send +any virtual RPCs. + +The client can start sending virtual RPCs immediately, before the initial +metadata / payload reach the server. However, the server application will +provide an explicit signal (a "barrier") telling the server session handler that +all common application context has been successfully set up, and the server is +ready to accept virtual RPC traffic. Internally, this signal triggers sending +the `server_initial_metadata` from the server to the client. + +While the client application does not explicitly need to wait for this +signal, all virtual RPCs sent prior to the barrier will be queued internally on +the server side until this handshake is complete. This queuing is desirable on +the server side instead of the client side, to avoid adding an extra round trip +before sending the virtual requests, since this adds extra latency to all the +early virtual requests in the session. This latency penalty can be especially +bad for short-lived sessions, where clients might establish some session to +exchange very few requests before closing. + +Alternatively, while by default the application can send the virtual RPCs before +the session is acknowledged, the client may also strictly want to wait for the +session to be established correctly. Since the server will send the +`server_initial_metadata` after session acknowledgement, the client can wait +for this signal (e.g., via an `OnSessionAcknowledged` callback) to validate +session acknowledgement before sending virtual RPCs. + +```cpp +void ClientReactor::OnSessionReady(grpc::internal::Call call) override { + channel_ = grpc:::CreateVirtualChannel(std::move(call)); +} + +void ClientReactor::OnSessionAcknowledged(bool ok) override { + if (ok) { + // Start sending vRPCs + } else { + // Shutdown + } +} +``` + +#### Deadlines and Cancellations + +We must support deadlines and cancellations for both the session and +each individual virtual request. + +* **Session Deadline**: The session deadline is converted to a `grpc-timeout` +header and sent to the server as the standard initial metadata. +When this expires, the gRPC client cancels the Session Request with a +`DeadlineExceeded` status immediately. The session handler then considers all +live virtual requests cancelled. Note that the error status on the virtual RPC +will not be `DeadlineExceeded`, since the virtual RPC's deadline did not expire. +Instead, the virtual RPC will fail with `Unavailable` error status, indicating +an underlying connection unavailability. + +* **Virtual RPC Deadline**: Deadlines on individual vRPCs work natively, same as +non-virtual RPCs. The only difference will be that, when the deadline expires, +the cancellation signal will be caught by the session handler and sent as a +payload message over the physical stream. + +#### Client Retries + +For regular RPCs, gRPC natively supports retries. Clients can define their retry +policy in the service config provided to the channel arguments. A retry policy +allows the users to define the following parameters: + +* **maxAttempts**: how many times to attempt the RPC before failing. +* **backOff**: configures delay between attempts. +* **retryableStatusCodes**: retry only when receiving these status codes. + +If retries are enabled for Bi-directional streams, client maintains a list of +all messages on the stream, until the initial metadata is received from the +server. In case of rejection, all these messages are retried. + +Retries for the session establishment request itself will work exactly the same +as regular RPCs, using the same retry configuration. Because +`server_initial_metadata` is not sent until the server application's +acknowledgement, all queued virtual RPCs will also be safely retried on the new +physical stream. + +For virtual RPC failures, retries on the same session are +**generally not advisable**, because retries should typically happen to +different backends. But since the session is tied to a single specific backend, +retries should ideally be handled by the client application itself. +That said, configuration options to retry on virtual channels will be provided +for clients who explicitly require it. + +#### Graceful Shutdown + +The server can initiate a graceful shutdown of the session, telling the client +to finish all scheduled virtual RPCs but not start new ones. + +* Internally, gRPC will accomplish this by sending an HTTP/2 `GOAWAY` frame on +the inner transport. +* Upon receiving this, the client virtual channel enters the +`GRPC_CHANNEL_TRANSIENT_FAILURE` state. New virtual RPCs are immediately +rejected with `UNAVAILABLE` status. +* Once all pending virtual RPCs complete, the session will be cleanly shut down. + +### Data Flow and Multiplexing + +#### Wire Format: HTTP/2 on HTTP/2 + +![HTTP/2 double stack](A116_graphics/vrpc2.png) + +To natively inherit gRPC's rich feature set, we will build the virtual stack as +a new HTTP/2 transport stack, running *over* the standard HTTP/2 stack. + +1. The client application first initiates a physical stream over the primary +HTTP/2 transport. The initial metadata and the first payload message are sent +as is over this stream. +2. This established stream is wrapped in a new gRPC Endpoint. We then create the +new inner HTTP/2 transport and the virtual channel with this endpoint, and +return this virtual channel to the client application. +3. Virtual RPCs on this channel are first sent to the inner HTTP/2 transport, +the transport encodes these into raw bytes and writes to the Endpoint. +These raw bytes are then sent as payload messages over the outer stream wrapped +in this endpoint. + +4. On the server side also, receiving a session request similarly spins up an +inner HTTP/2 transport that decodes the subsequent payload messages as +independent inner streams. + +#### Alternative Wire Format + +An alternative design considered was defining a custom frame format over the +physical stream where the over the wire payload message would be structured as +follows: +1. **Metadata Length (4 Bytes)**: An integer denoting the size of the following serialized metadata. +2. **Serialized Metadata**: An encapsulated frame +(`ClientVrpcFrame` or `ServerVrpcFrame`) containing the unique virtual RPC ID +and the frame type (initial metadata, payload, cancellation, or half-close). +3. **Optional Raw Payload**: The raw binary bytes of the application message +follow immediately after the serialized proto. This is present only if the +message is of the payload type. + +#### Why HTTP/2 on HTTP/2 is chosen + +The big benefit of the HTTP/2 on HTTP/2 approach is that we do not need to +re-invent many gRPC features for the virtual stack, which are already provided +by the HTTP/2 stack. + +* If we utilized the alternative wire format, we would have to implement many +core features like Flow Control, max_concurrent_streams, message chunking, +retries, etc. By using the HTTP/2 as the inner wire format, we get all these +features for free. + +* The downside is that this wire format is somewhat more complicated, and there +is a non-trivial cost to the double encoding. + +### Flow Control + +In a traditional bidirectional stream, if an application handler is slow and +fails to read messages, gRPC’s internal buffers fill up, eventually causing the +receiver to push back on the sender and stalling the entire stream. +When multiple independent virtual RPCs share a single physical stream, a single +slow handler could potentially stall all other concurrent virtual RPCs in the +session. + +The HTTP/2-on-HTTP/2 architecture implicitly addresses this through: + +* **Stall Prevention**: The outer HTTP/2 transport proactively writes data to +the inner transport's endpoint. This frees the outer stream to continue +receiving and processing frames for other virtual RPCs, even if one specific +vRPC is being handled slowly. +* **Backpressure**: The inner transport’s flow control window and +`max_concurrent_streams` setting prevent unbounded queue growth. +* **Fairness**: Native HTTP/2 frame interleaving ensures that large virtual +RPC payloads do not block smaller, latency-sensitive ones. + +### Server-Side Architecture: Dual Server Approach + +Often, gRPC servers are configured with mandatory interceptors or modules +(e.g., authentication, authorization) that run on every incoming connection. +Because virtual RPCs have already been authorized at the outer session level, +re-running these modules on the inner virtual RPCs is unnecessary. + +To allow virtual handlers to bypass these mandatory checks, the design uses a +dual-server approach. We will create a new gRPC server without these modules. +Virtual service handlers are registered exclusively on this second server. +When the primary gRPC server receives a Session Request, it bridges the inner +HTTP/2 transport to this secondary server, ensuring vRPCs seamlessly bypass the +primary server's mandatory modules. + +This also provides an easy interface to configure the inner HTTP/2 server +differently. Any configuration we want specific to the inner transport, can just +be set on this second server. + +### C++ User-Facing APIs + +The C++ prototype uses Reactor-based APIs to manage the session lifecycle and +establish virtual channels using the generic API approach. + +#### Client-Side API + +Clients establish a session using `ClientSessionReactor` and +`GenericStubSession`. + +```cpp +class MySessionReactor : public grpc::ClientSessionReactor { + public: + void OnSessionReady(grpc::internal::Call call) override { + // Create the virtual channel using the established call. + // CreateVirtualChannel creates a light-weight client side channel. + virtual_channel_ = grpc::experimental::CreateVirtualChannel(std::move(call)); + } + void OnSessionAcknowledged(bool ok) override { /* Session is fully ready */ } + void OnDone(const grpc::Status& s) override { /* Session ended */ } + + private: + std::shared_ptr virtual_channel_; +}; + +// Setting up the session +grpc::experimental::GenericStubSession session_stub(channel); +auto* reactor = new MySessionReactor(); +session_stub.PrepareSessionCall(&context, "/service/SessionRequest", {}, &request, reactor); +reactor->StartCall(); +``` + +#### Server-Side API + +On the server, `ServerSessionReactor` is used to establish the common session +context, and represent the lifecycle of the session. + +```cpp +class MyServerSessionReactor : public grpc::ServerSessionReactor { + public: + MyServerSessionReactor() { + // Notify gRPC that we are ready to accept virtual RPCs + StartVirtualRPCs(); + } + void TriggerGracefulShutdown() { + InitiateGracefulShutdown([this](absl::Status status) { + Finish(grpc::Status::OK); + }); + } + void OnDone() override { delete this; } +}; +``` + +#### Context Propagation + +Applications need to prepare and share session-level resources (such as +authentication credentials or cached application states) among multiple virtual +RPC handlers. To provide easy and implicit lifetime management for these +session-level resources, this application context will be stored on the Session +Call's **Arena**. + +Because gRPC’s Arena is a ref-counted object, when a virtual call is created, +gRPC can simply store a reference to the parent session call's arena on the new +virtual call's arena. + +This design ensures that: + +* **Decoupled Lifetimes**: Shared context remains valid even if the parent +session handler completes, as the virtual handlers hold a reference to the same underlying memory. +* **Implicit Management**: Resources are automatically cleaned up only after +both the session and all its associated virtual RPCs have finished. + +Users can set and retrieve this shared context through the `ServerContext` API: + +```cpp +// In Session Handler: Set the context on the outer request ServerContext +context->SetContext(std::make_shared(...)); + +// In Virtual RPC Handler: Retrieve the shared application state from the session +auto app_context = context->GetSessionContext(); +``` + +### Java User-Facing APIs + +TBD + +### Go User-Facing APIs + +TBD + diff --git a/A116_graphics/vrpc2.png b/A116_graphics/vrpc2.png new file mode 100644 index 000000000..4493f88fa Binary files /dev/null and b/A116_graphics/vrpc2.png differ diff --git a/A116_graphics/vrpc_flow.png b/A116_graphics/vrpc_flow.png new file mode 100644 index 000000000..62e8d31c9 Binary files /dev/null and b/A116_graphics/vrpc_flow.png differ