Add routing microbenchmark for choose_replica + dispatch pattern

[jeffreywang-anyscale]

Description

Following up on #63255 (comment), we'd like to show the delta between choose_replica + dispatch and remote in our DB dashboards.

Release test results -- latencies in ms (https://buildkite.com/ray-project/release/builds/92517/canvas?sid=019e1a3c-642f-4a42-867b-2818577272b6&tab=output)

Percentile	remote	choose_replica + dispatch	Delta
p50	1.040	1.784	+0.74
p90	1.149	1.922	+0.77
p95	1.199	1.970	+0.77
p99	1.437	2.137	+0.70

Related issues

Link related issues: "Fixes #1234", "Closes #1234", or "Related to #1234".

Additional information

Optional: Add implementation details, API changes, usage examples, screenshots, etc.

[jeffreywang-anyscale]

Kicking off release tests to populate data to verify databricks query.

[gemini-code-assist]

Code Review

This pull request introduces a new benchmarking mode, 'choose_dispatch', to evaluate the latency of the choose_replica and dispatch pattern in Ray Serve. The changes include adding a new benchmarking method to the Benchmarker class, updating the handle_noop_latency script with a new CLI option, and expanding the microbenchmark workloads to include this new mode. A review comment suggests refactoring the run_latency_benchmark method to avoid duplicating the definition of the internal benchmark function, which would improve code maintainability.

[gemini-code-assist]

The current implementation defines the local function f twice within different branches of the if statement. This logic can be refactored to be more concise and maintainable by assigning the target method to a variable first, and then defining f once.

        if mode == "remote":
            func = self.do_single_request
        elif mode == "choose_dispatch":
            func = self.do_single_choose_dispatch
        else:
            raise ValueError(f"Unknown mode {mode!r}")

        async def f():
            await func(payload)