Feature Request: Add option for isolated connections for Pulsar consumers and producers

[lhotari]

Motivation

Currently, in Pulsar, TCP connections are shared across all consumers and producers for a client. This can lead to issues with backpressure and flow control, particularly for producers. The shared connection usage causes head-of-line blocking problems. To address this, it is proposed to add configuration options to allow consumers and producers to use isolated TCP/IP connections or connection groups.

• Allow configuring individual consumers or producers to use a dedicated, non-shared TCP connection (or connection group) to the broker.
• Provide an option for multi-partition producers to allow sharing of the dedicated connections for all partitions of that producer instance.
• Implement similar isolation options for consumers.
• This feature would complement the proposed throttling-based flow control solution in "PIP-385: Add rate limit semantics to Pulsar protocol and Java Client" [improve][pip] PIP-385 Add rate limit semantics to pulsar protocol and Java Client #23398, providing an additional mechanism to improve connection management and reduce interference between different producers/consumers.
• The isolation options would be configurable through the client API, allowing developers to fine-tune connection usage based on their specific requirements.

Benefits

• Reduced interference between producers/consumers sharing connections
• More predictable performance for high-priority or high-volume producers/consumers
• Easier troubleshooting of connection-related issues

This feature provides a simple improvement in connection management and performance isolation, which can be implemented as an interim solution or alongside more comprehensive flow control mechanisms such as PIP-385.

Additional Context

• "PIP-385: Add rate limit semantics to Pulsar protocol and Java Client" [improve][pip] PIP-385 Add rate limit semantics to pulsar protocol and Java Client #23398

[lhotari]

@grssam Thanks for your work on PIP-385 to improve rate limiting and flow control. Have you considered adding an option for isolated connections as well? This has been discussed several times in relation to rate limiters and could complement PIP-385 nicely.
While PIP-385 will significantly improve the "head-of-line blocking" problem by adding rate limit semantics to the protocol, it won't completely eliminate the queuing that inherently occurs in a TCP connection. For certain high-priority or latency-sensitive use cases, having the ability to isolate producers or consumers on dedicated connections could be valuable.
I understand there's a tradeoff with increased connection count, but for some scenarios, that may be an acceptable cost for improved isolation and predictability. WDYT?

[grssam]

@grssam Thanks for your work on PIP-385 to improve rate limiting and flow control. Have you considered adding an option for isolated connections as well? This has been discussed several times in relation to rate limiters and could complement PIP-385 nicely. While PIP-385 will significantly improve the "head-of-line blocking" problem by adding rate limit semantics to the protocol, it won't completely eliminate the queuing that inherently occurs in a TCP connection. For certain high-priority or latency-sensitive use cases, having the ability to isolate producers or consumers on dedicated connections could be valuable. I understand there's a tradeoff with increased connection count, but for some scenarios, that may be an acceptable cost for improved isolation and predictability. WDYT?

For specific known high priority topics, can't the users simply create separate client? my PIP tries to solve the overall problem holistically in a default situation along with adding the protocols to notify the clients.. .. but for certain specific limited sub-set which has higher priority and needs separate connection, it can already be solved today..

[lhotari]

For specific known high priority topics, can't the users simple not create separate client?

@grssam Good point. Yes, that's possible. There's a proposal to make it easier "PIP-234: Support using shared thread pool across multiple Pulsar client instance" #19074. That would be an alternative solution. I think it would be good to finally implement PIP-234.

my PIP tries to solve the overall problem holistically in a default situation.. but for certain specific limited sub-set it can already be solved today..

Yes that's true. I'm not saying that isolation would be a replacement for PIP-385. We need a producer flow control solution in the Pulsar binary protocol. That has been in discussions for a very long time and I appreciate that you are driving the efforts to finally act and make it happen.

[grssam]

For specific known high priority topics, can't the users simple not create separate client?

@grssam Good point. Yes, that's possible. There's a proposal to make it easier "PIP-234: Support using shared thread pool across multiple Pulsar client instance" #19074. That would be an alternative solution. I think it would be good to finally implement PIP-234.

Sorry, I updated my original reply a bit later. I realized i was doing double negatives. what I meant was actually that for higher priority topics, users can always create a separate client instead of using the client that spins up producers for the rest of the topics

[lhotari]

Current workaround would be to create a Pulsar client for each producer if there's need to isolate different producers in an application that produces to multiple topics or has both producers and consumers. PIP-234 org.apache.pulsar.client.api.PulsarClientSharedResources added in #24784 and #24790 makes it efficient to have a larger number of Pulsar client instances in a single application. However, this workaround wouldn't address partitioned topic producers and consumers with complete isolation.
Another workaround would be to increase the number of client connections per broker, however that doesn't completely eliminate the issue.