feat: add vLLM-Omni EC2 and SageMaker DLC images

[Yadan-Wei]

vLLM-Omni DLC: EC2 and SageMaker Deep Learning Containers for Omni-Modality

Models

Summary

Adds vLLM-Omni DLC images for EC2 and SageMaker on Amazon Linux 2023, enabling
serving of omni-modality models (TTS, image generation, video generation,
multimodal chat) via vllm-omni==0.18.0.

What's Included

Dockerfile (docker/vllm/Dockerfile.amzn2023)

• New stages: omni-deps, builder-oss-omni, omni-base, vllm-omni-ec2-amzn2023,
  vllm-omni-sagemaker-amzn2023
• Installs vllm-omni==0.18.0 via pip on top of vLLM runtime
• SPAL system deps: espeak-ng, sox, ffmpeg-free
• Pre-built runtime base support (RUNTIME_BASE arg) to skip vLLM compile in PR
  builds (~120min → ~2min)

Entrypoints

• omni_dockerd_entrypoint.sh — EC2: exec vllm serve --omni "$@"
• omni_sagemaker_entrypoint.sh — SageMaker: parses SM_VLLM_* env vars, adds
  --middleware for routing

SageMaker Routing Middleware (omni_sagemaker_serve.py)

• ASGI middleware injected via --middleware flag (single process, no proxy)
• Routes /invocations to the correct vllm-omni endpoint based on
  CustomAttributes: route=
• Falls through to vLLM's built-in /invocations handler for chat/completion/
  embed
• Supports: /v1/audio/speech, /v1/images/generations, /v1/videos,
  /v1/chat/completions
• 12 unit tests covering routing, fallthrough, adapter coexistence

CI Configs & Workflows

• vllm-omni-ec2-amzn2023.yml, vllm-omni-sagemaker-amzn2023.yml — framework
  configs
• pr-vllm-omni-ec2-amzn2023.yml, pr-vllm-omni-sagemaker-amzn2023.yml — PR
  workflows with build-runtime caching
• reusable-vllm-omni-model-tests.yml — generic smoke test workflow supporting
  JSON and multipart/form-data
• vllm-omni-model-tests.yml — per-model test config with route, request
  payload, and validation

Smoke Tests (5 models, 4 routes)

Model	Route	Runner
Qwen3-TTS-1.7B-CustomVoice	/v1/audio/speech	g6xl
FLUX.2-klein-4B	/v1/images/generations	g6xl
Wan2.1-T2V-1.3B	/v1/videos	g6e4xl
Qwen2.5-Omni-3B	/v1/chat/completions	g6e12xl

• EC2 tests use real entrypoint, hit OpenAI-compatible API directly
• SageMaker tests use real entrypoint, hit /invocations with middleware
  routing
• Container log dump (500 lines) on failure
• Orphaned endpoint cleanup step

SageMaker Endpoint Tests

• Sync endpoint test with retry for torch.compile warmup (60s SageMaker
  timeout)
• Async endpoint test using AsyncInferenceConfig (bypasses 60s limit)

S3 Model Cache

• Models pre-cached in s3://dlc-cicd-models/omni-models/ as tar.gz
• Uses download-model action with ETag-based caching and flock-based
  concurrency

Key Design Decisions

• Middleware over proxy: Single-process ASGI middleware (--middleware flag
  ) instead of a separate proxy process. Reuses vLLM's existing /invocations,
  /ping, /health handlers.
• Per-model test config: Route, request payload, content type, and
  validation defined in YAML. Adding a new model = adding a config entry, no
  code changes.
• Pre-built runtime base: build-runtime job checks ECR for cached runtime
  image, builds only on first run per vLLM version. Subsequent PR builds skip
  the ~120min compile.

Testing

• Middleware unit tests (12 tests)
• EC2 smoke tests (4 models)
• SageMaker smoke tests (4 models)
• SageMaker endpoint test (sync + async)
• Pre-commit checks
• Middleware verified on upstream vllm/vllm-omni:v0.18.0 Docker Hub image

---

Toggle if you are merging into master Branch

By default, docker image builds and tests are disabled. Two ways to run builds and tests:

1. Using dlc_developer_config.toml
2. Using this PR description (currently only supported for PyTorch, TensorFlow, vllm, and base images)

How to use the helper utility for updating dlc_developer_config.toml

Assuming your remote is called origin (you can find out more with git remote -v)...

• Run default builds and tests for a particular buildspec - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -cp origin

• Enable specific tests for a buildspec or set of buildspecs - also commits and pushes changes to remote; Example:

python src/prepare_dlc_dev_environment.py -b </path/to/buildspec.yml> -t sanity_tests -cp origin

• Restore TOML file when ready to merge

python src/prepare_dlc_dev_environment.py -rcp origin

NOTE: If you are creating a PR for a new framework version, please ensure success of the local, standard, rc, and efa sagemaker tests by updating the dlc_developer_config.toml file:

• sagemaker_remote_tests = true
• sagemaker_efa_tests = true
• sagemaker_rc_tests = true
• sagemaker_local_tests = true

How to use PR description

Use the code block below to uncomment commands and run the PR CodeBuild jobs. There are two commands available:

• # /buildspec <buildspec_path>
  • e.g.: # /buildspec pytorch/training/buildspec.yml
  • If this line is commented out, dlc_developer_config.toml will be used.
• # /tests <test_list>
  • e.g.: # /tests sanity security ec2
  • If this line is commented out, it will run the default set of tests (same as the defaults in dlc_developer_config.toml): sanity, security, ec2, ecs, eks, sagemaker, sagemaker-local.

# /buildspec <buildspec_path>
# /tests <test_list>

Toggle if you are merging into main Branch

PR Checklist

• [] I ran pre-commit run --all-files locally before creating this PR. (Read DEVELOPMENT.md for details).

[junpuf]

we'll use the save repo name "vllm" instead of creating new repo

[Yadan-Wei]

Will update this section when we have a real prod image.