Skip to content

[serve][llm] Add direct streaming target for LLM tests#63471

Closed
eicherseiji wants to merge 1 commit into
ray-project:masterfrom
eicherseiji:seiji/direct-streaming-llm-test-variant
Closed

[serve][llm] Add direct streaming target for LLM tests#63471
eicherseiji wants to merge 1 commit into
ray-project:masterfrom
eicherseiji:seiji/direct-streaming-llm-test-variant

Conversation

@eicherseiji
Copy link
Copy Markdown
Contributor

@eicherseiji eicherseiji commented May 18, 2026

Description

Adds a direct-streaming variant of the existing Ray LLM OpenAI compatibility test target.

  • Duplicates serve/gpu/integration/test_openai_compatibility.py at the Bazel level with a _direct_streaming target suffix.
  • Sets RAY_SERVE_LLM_ENABLE_DIRECT_STREAMING=1 only on the duplicated target.
  • Reuses the same Python test file for the normal and direct-streaming runs.
  • Skips the existing multi-model no-model-parameter case when direct streaming is enabled, since direct streaming currently supports exactly one LLM config.
  • Adds minimal direct ASGI support to MockVLLMEngine so the existing OpenAI compatibility tests can run through the direct-streaming path without a real vLLM frontend.

Related issues

Related to #63167.

Additional information

Validation run locally:

python python/ray/setup-dev.py -y -a llm
python -m py_compile \
  python/ray/llm/tests/serve/gpu/integration/test_openai_compatibility.py \
  python/ray/llm/tests/serve/mocks/mock_vllm_engine.py
python -m pytest --collect-only -q \
  python/ray/llm/tests/serve/gpu/integration/test_openai_compatibility.py
uvx pre-commit run --files \
  python/ray/llm/tests/BUILD.bazel \
  python/ray/llm/tests/serve/gpu/integration/test_openai_compatibility.py \
  python/ray/llm/tests/serve/mocks/mock_vllm_engine.py
git diff --check

The duplicated Bazel target should be exercised in Buildkite.

@gemini-code-assist
Copy link
Copy Markdown
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Signed-off-by: Seiji Eicher <seiji@anyscale.com>
@eicherseiji eicherseiji force-pushed the seiji/direct-streaming-llm-test-variant branch from 798ca0e to fa6dd2b Compare May 18, 2026 21:03
@eicherseiji eicherseiji changed the title [serve][llm] Add direct streaming OpenAI compatibility smoke tests [serve][llm] Run OpenAI compatibility tests with direct streaming May 18, 2026
@eicherseiji eicherseiji added the go add ONLY when ready to merge, run all tests label May 18, 2026
@eicherseiji eicherseiji changed the title [serve][llm] Run OpenAI compatibility tests with direct streaming [serve][llm] Add direct streaming OpenAI compatibility test target May 18, 2026
@eicherseiji eicherseiji changed the title [serve][llm] Add direct streaming OpenAI compatibility test target [serve][llm] Add direct streaming target for LLM tests May 18, 2026
@eicherseiji
Copy link
Copy Markdown
Contributor Author

Closing as superseded: the direct-streaming OpenAI compatibility target has been folded into #63468.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

go add ONLY when ready to merge, run all tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant