[refactor] Remove legacy inference codepath (#1835)

# What does this PR do?

## Summary

This PR removes the old Ray actor-based inference stack and standardizes
SkyRL on the HTTP/vLLM-based inference path. The
`_SKYRL_USE_NEW_INFERENCE` flag is removed entirely, the legacy
`inference_engines/` module is deleted, and the remaining training,
generation, examples, and docs surfaces now target the HTTP/vLLM server
path.

The PR also migrates several examples and integrations that still
assumed the old path, fixed/static inference ports, or the legacy custom
vLLM server module.

## Changes made

### Core

#### Legacy inference stack removal

- Deleted the legacy `skyrl/backends/skyrl_train/inference_engines/`
package, including the old Ray-wrapped inference client, remote
inference engine, HTTP endpoint client, vLLM engine wrapper, and old
custom vLLM server.
- Deleted the old `inference_servers/vllm_worker.py` wrapper.
- Removed `_SKYRL_USE_NEW_INFERENCE` and collapsed all old/new inference
branches onto the HTTP/vLLM path.
- Removed old-inference-only tests, scripts, and CI workflow files.
- Moved the shared inference interface and I/O types into
`inference_servers/base.py`.
- Moved reusable inference helper logic into
`inference_servers/engine_utils.py`.

#### Inference interface and remote client

- Simplified `InferenceEngineInterface` around the methods actually used
by the HTTP/vLLM path.
- Made `RemoteInferenceClient` implement the interface directly so the
interface and concrete client cannot drift.
- Added interface methods/properties used by generators and
integrations:
  - `model_name`
  - `get_endpoint_url()`
  - `get_world_size()`
  - `render_chat_completion()`
  - `finish_session()`
- Updated generators and integrations to use `InferenceEngineInterface`
and `get_endpoint_url()` instead of reaching into
`RemoteInferenceClient.proxy_url` when possible.
- Added coverage that `served_model_name` is passed into vLLM and used
correctly by `RemoteInferenceClient` and server setup.

#### vLLM server entrypoint

- Extended
`skyrl.backends.skyrl_train.inference_servers.vllm_server_actor` so it
can also be launched as a standalone module:
- `python -m
skyrl.backends.skyrl_train.inference_servers.vllm_server_actor`
- The standalone entrypoint uses vLLM's OpenAI server setup while
preserving SkyRL's custom endpoints needed by training and weight sync.
- Added GPU coverage for the standalone entrypoint:
  - health check
  - OpenAI chat completion
  - SkyRL token-in/token-out generation endpoint
- This lets external rollout-server deployments use the same SkyRL-aware
vLLM server path as the Ray actor path.

#### Weight sync cleanup

- Removed legacy receiver-side abstractions and the unused
`WeightLoader`.
- Simplified weight transfer strategy contracts around the new
vLLM-native receiver path.
- Weight loading on the inference side is handled through vLLM's native
transfer engines + `NewInferenceWorkerWrap`.SkyRL's
`WeightTransferReceiver` abstractions have been removed for now.
- Added strategy-to-vLLM transfer engine mapping:
  - broadcast/NCCL -> `NCCLWeightTransferEngine`
  - CUDA IPC -> `IPCWeightTransferEngine`
- Simplified `BroadcastTransferStrategy.create_init_info` by requiring
the inference world size from `client.get_world_size()`.
- Removed stale legacy chunk-transfer branches and related tests.

#### Config and routing cleanup

- Removed old fixed HTTP endpoint config knobs:
  - `generator.inference_engine.http_endpoint_host`
  - `generator.inference_engine.http_endpoint_port`
  - `generator.inference_engine.enable_http_endpoint`
- Cleaned up `generator.inference_engine.async_engine` references across
docs and examples. The config loader now tolerates/removes legacy
`async_engine=true` overrides but rejects unsupported
`async_engine=false`.
- Removed old helper code that only existed for the legacy path,
including `route_prompts_to_engines` and `hash_with_sha256`.
- Updated SkyRL Agent integration code to stop depending on the removed
HTTP endpoint config and old dataloader workarounds.

#### Tinker updates

- Updated `SkyRLTrainBackend` so colocated inference engines are put to
sleep after initialization, matching the colocated behavior with the
python entrypoint and freeing memory before training work starts.
- Updated Tinker E2E scripts to pin the `tinker-cookbook` commit for
reproducibility.
- Updated Tinker/SkyRL sample routing and related tests to work through
the HTTP/vLLM path.

### Examples

#### MiniSWE Agent example

- Migrated `MiniSWEAgentGenerator` to the new inference interface.
- The generator now gets the router URL from
`inference_engine_client.get_endpoint_url()`.
- `OPENAI_BASE_URL` is set at runtime inside the Ray task that runs
MiniSWE-Agent, because LiteLLM reads that environment variable from the
task process.
- Removed the fixed `OPENAI_BASE_URL` assumption from `.env.miniswe`;
the router port is dynamic in the new inference path.
- Updated MiniSWE launch scripts to remove legacy inference config
overrides.

#### Thunder-Agent example

- Migrated the Thunder-Agent R2EGym 32B recipe away from the legacy vLLM
server module.
- The rollout server launcher now defaults to SkyRL's new standalone
vLLM server entrypoint:
  - `skyrl.backends.skyrl_train.inference_servers.vllm_server_actor`
- This is intentionally not the vanilla
`vllm.entrypoints.openai.api_server`, because SkyRL still needs custom
endpoints for weight sync and inference-server control.
- Updated the Thunder-Agent scripts and README around the new
rollout-server path.
- Added robust rollout-server process-tree cleanup in the launcher so
external vLLM EngineCore / worker subprocesses do not leak GPUs after
shutdown.
- Updated Thunder-Agent generator/router handling for the new endpoint
semantics and program-release flow.

#### Remote inference server examples

- Migrated the old `remote_inference_engine` example into
`remote_inference_server`.
- Added a serve/vLLM-native server launch script for remote inference.
- Updated remote inference docs and examples to point at the serve
entrypoint and HTTP/vLLM server path.

#### Harbor, Verifiers, RLM, and OpenRouter examples

- Updated Harbor and Verifiers generators to use `get_endpoint_url()`
instead of removed fixed host/port config.
- Updated RLM/OpenRouter client code for interface parity with
`InferenceEngineInterface`.
- Removed old `enable_http_endpoint` / `async_engine` overrides from
Harbor, Verifiers, OpenEnv/OpenReward, RLM, and related integration
scripts.

#### General training scripts

- Removed obsolete `generator.inference_engine.async_engine=true` and
old HTTP endpoint overrides from the example training scripts.
- Migrated the algorithm, Megatron, LoRA, GSM8K, text-to-SQL, search,
VisGym, fully async, router replay, and integration launch scripts to
the new default inference behavior.
- Updated Tinker example server scripts to use the current serve/vLLM
path.

#### Removed obsolete examples

- Removed the old FlashRL example and docs that depended on the old
inference stack.
- Removed old helper scripts that compared old-vs-new inference or
launched multiple legacy remote servers.

### Docs

#### Architecture and configuration docs

- Updated the inference architecture docs to describe the HTTP/vLLM path
as the standard path.
- Removed references to `_SKYRL_USE_NEW_INFERENCE`, the old Ray actor
inference stack, and deleted legacy classes.
- Updated configuration docs to remove the deprecated HTTP endpoint
host/port, `enable_http_endpoint`, and `async_engine` knobs.
- Updated placement and Tinker architecture docs for the new colocated
inference behavior.

#### Example docs

- Updated example pages for PPO, Geometry3K, Search, VisGym, multi-turn
text-to-SQL, remote inference, and fully async training.
- Removed the FlashRL example page and metadata entry.
- Updated Harbor and agent-integration docs to use the new inference
endpoint behavior.

#### Internal contributor docs

- Updated `.claude/docs` architecture, inference, contributing, and
weight-sync notes to match the new inference and weight-sync structure.
- Updated API/sidebar metadata after deleting or moving example pages.

### Tests and CI cleanup

- Deleted legacy-only tests under
`tests/backends/skyrl_train/inference_engines/`.
- Removed old GPU CI workflow/scripts for the legacy inference stack.
- Updated shared tests to run directly against the new inference path.
- Added GPU coverage for the standalone SkyRL vLLM server entrypoint.
- Added/updated coverage for:
  - `RemoteInferenceClient`
  - `served_model_name`
  - prompt/chat rendering
  - VLM generation
  - LoRA serving/reload
  - weight sync
  - generator endpoint routing
  - Tinker sample routing

## Validation

- `ruff` and `black` pass.
- CPU test coverage passes for weight sync, transfer strategies, prompt
routing, generators, trainer, config migration, and Tinker sample
routing.
- GPU CI workflows pass locally:
  - SkyRL-Train-GPU
  - SkyRL-Train-Megatron-GPU
  - Megatron-Model-GPU-CI
  - SkyRL-GPU-E2E-CI
  - H100 GPU CI
  - SkyRL-GPU-E2E-CI-Tinker-Fully-Async sanity run

---------

Signed-off-by: SumanthRH <sumanthrh99@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: SumanthRH

.claude/docs/architecture.md

@@ -10,8 +10,7 @@ skyrl/                  # Core library
 ├── backends/           # Backend implementations
 │   └── skyrl_train/    # FSDP/Megatron training backend
 │       ├── distributed/        # Dispatch, FSDP/Megatron strategies
-│       ├── inference_engines/  # Legacy inference path
-│       ├── inference_servers/  # New inference path
+│       ├── inference_servers/  # HTTP inference path (RemoteInferenceClient, vLLM servers, router)
 │       ├── weight_sync/        # Weight extraction and transfer
 │       └── workers/            # FSDP/Megatron workers
 ├── train/              # Training entrypoints, config, dataset, generators, trainer

.claude/docs/contributing.md

@@ -57,7 +57,7 @@ Do:
 
 Don't:
 
-"Uses the start/update/finish lifecycle to enable chunked transfers. Per chunk, all tensors are packed into a single contiguous CUDA buffer (one dtype per chunk, guaranteed by the weight extractor) and one IPC handle is created for the packed buffer per rank. This mirrors the legacy path and avoids the one-handle-per-param ceiling of vLLM's default IPCWeightTransferEngine, which otherwise dominates latency for models with many small parameters."
+"Uses the start/update/finish lifecycle to enable chunked transfers. Per chunk, all tensors are packed into a single contiguous CUDA buffer (one dtype per chunk, guaranteed by the weight extractor) and one IPC handle is created for the packed buffer per rank. This avoids the one-handle-per-param ceiling of vLLM's default IPCWeightTransferEngine, which otherwise dominates latency for models with many small parameters."
 
 
 Do:

.claude/docs/inference.md

@@ -1,14 +1,14 @@
 # Inference
 
-For training-to-inference weight transfer (`WorkerWrap`, broadcast vs. CUDA IPC, lifecycle), see [`weight_sync.md`](weight_sync.md).
+For training-to-inference weight transfer (`NewInferenceWorkerWrap`, broadcast vs. CUDA IPC, lifecycle), see [`weight_sync.md`](weight_sync.md).
 
 ## Architecture
 
 - Key abstractions: `RemoteInferenceClient` , `ServerGroup`, `VLLMServerActor`, `VLLMRouter`
 - `RemoteInferenceClient` interacts with HTTP endpoints: 
     - **Data plane**: Interact with router for completions requests.
     - **Control plane**: Fan-out to individual server URLs for weight sync, pause/resume.
-- This is the new inference codepath enabled by default (`_SKYRL_USE_NEW_INFERENCE=1`)
+- Shared inference interfaces and types live in `inference_servers/base.py` (`InferenceEngineInterface`, `InferenceEngineInput`/`Output`, `ConversationType`); shared helpers (`build_engine_runtime_env`, `get_sampling_params_for_backend`) live in `inference_servers/engine_utils.py`.
 
 ## vLLM Router
 
@@ -35,19 +35,3 @@ All under `generator.inference_engine.*`:
 ## Placement
 - Colocated: vLLM and training workers (FSDP/Megatron) are placed on the same set of GPUs. We offload/backload each component as needed. During weight syncing, model weights from vLLM as well as model weights from the training workers remain on GPU
 - Non-colocated: vLLM and training workers (FSDP/Megatron) are placed on a different set of GPUs. This reduces the number of available GPUs per component by half, but is in fact the preferred setup for agentic RL with SkyRL. This is because non-colocated setups allow for asynchronous training, where training and inference can progress together. Inference is typically dominated by a long tail of stragglers, and is also typically the time consuming component, and thus using half the number of GPUs doesn't affect inference time for a batch as much.
-
-# Legacy Inference Codepath
-
-`_SKYRL_USE_NEW_INFERENCE=0` triggers the legacy inference codepath, in `skyrl/backends/skyrl_train/inference_engines/`.
-
-## Architecture
-
-- Key abstractions: `InferenceEngineInterface`, `InferenceEngineClient`, `RemoteInferenceEngine`, `RayWrappedInferenceEngine`, `VLLMRayActor` / `AsyncVLLMRayActor`.
-- `InferenceEngineClient` (`inference_engine_client.py`) holds a `List[InferenceEngineInterface]` and itself implements the interface, so callers talk to it as if it were one engine.
-    - **Routing**: prompts are sharded across engines by `route_prompts_to_engines` (respects `session_ids` for stickiness) — there is no `vllm-router`; the client does the fan-out.
-    - **Control plane** (`wake_up`, `sleep`, `init_weight_update_communicator`, `update_named_weights`, `reset_prefix_cache`, `pause_generation`, `resume_generation`, `teardown`) is fanned out to all engines via `_run_on_all_engines`.
-- Two engine implementations sit behind the interface:
-    - `RemoteInferenceEngine` (`remote_inference_engine.py`): HTTP client against an OpenAI-compatible server URL; weight sync goes through `RemoteWeightLoader` over the same HTTP endpoint.
-    - `RayWrappedInferenceEngine` (`ray_wrapped_inference_engine.py`): thin wrapper over a Ray `ActorHandle` (e.g. `VLLMRayActor` / `AsyncVLLMRayActor`) for in-cluster colocated engines.
-- Optional OpenAI-compatible HTTP endpoint in front of the client is spun up by `InferenceEngineClient._spin_up_http_endpoint` (`inference_engine_client_http_endpoint.py`) when `inference_engine_cfg.enable_http_endpoint=true`.
-

.claude/docs/weight_sync.md

@@ -9,34 +9,27 @@ Two-sided protocol with sender (training) / receiver (inference):
 ```
 skyrl/backends/skyrl_train/weight_sync/
 ├── base.py                 # WeightUpdateRequest, LoraLoadRequest, WeightChunk
-├── transfer_strategy.py    # WeightSyncInitInfo / Sender / Receiver / Strategy ABCs
+├── transfer_strategy.py    # WeightSyncInitInfo / Sender / Strategy ABCs (sender-side only; receive is vLLM-native)
 ├── broadcast_strategy.py   # NCCL broadcast (non-colocated)
 ├── cuda_ipc_strategy.py    # CUDA IPC (colocated)
 ├── weight_extractor.py     # Sharded-param -> dense tensor extraction
-├── weight_extractor_utils.py
-└── weight_loader.py        # WeightLoader ABC (sender-side driver)
+└── weight_extractor_utils.py
 ```
 
-vLLM worker-extension classes (loaded via `--worker-extension-cls`):
+vLLM worker-extension class (loaded via `--worker-extension-cls`):
 
-- `skyrl/backends/skyrl_train/inference_servers/vllm_worker.py` — `WorkerWrap`. **Legacy path.**  One or more calls to `load_weights(request)`.
-- `skyrl/backends/skyrl_train/inference_servers/new_inference_worker_wrap.py` — `NewInferenceWorkerWrap`. **New path** (`_SKYRL_USE_NEW_INFERENCE=1`, default). Three-phase chunked lifecycle.
+- `skyrl/backends/skyrl_train/inference_servers/new_inference_worker_wrap.py` — `NewInferenceWorkerWrap`. Three-phase chunked lifecycle.
+
+The weight sync implementation relies on the native vLLM weight sync APIs - `WeightTransferEngine` abstractions as well as native RPC endpoints for weight updates.
 
 ## Transfer Strategies
 
 - **Broadcast** (`BroadcastTransferStrategy`): NCCL collective. Used for **non-colocated** setups. Training and inference are on different GPUs; weights cross the wire over a dedicated process group.
 - **CUDA IPC** (`CudaIpcTransferStrategy`): Per-chunk packed buffer + one IPC handle per rank. Used for **colocated** setups (`colocate_all=true`). Both sides live on the same GPU; the receiver maps the sender's CUDA allocation directly.
 
-`WeightSyncInitInfo.strategy_type()` returns the receiver class. Strategy choice is decided by the sender and the receiver picks up the matching class via the pickled init info.
-
-## Lifecycle
-
-**Legacy path** (`WorkerWrap`):
-1. `init_weight_update_communicator(init_info)` — once per session. Constructs the receiver.
-2. `load_weights(request)` — per sync. Receives weights using strategy-specific weight receiver and loads weights into vLLM.
-3. `teardown_weight_receiver()` — on shutdown.
+Strategy choice is decided by the sender (`get_transfer_strategy_cls`). The init info is expanded per server via `for_servers()` / `to_api_payload()` and pushed to the servers through the HTTP control plane (`init_weight_update_communicator` → vLLM's native `/init_weight_transfer_engine`); the receive side is vLLM's native weight-transfer engine, driven by `NewInferenceWorkerWrap`.
 
-**New path** (`NewInferenceWorkerWrap`):
+## Lifecycle (`NewInferenceWorkerWrap`)
 1. `start_weight_update(is_checkpoint_format=True)` — initializes layerwise reload (moves layers to meta device, wraps loaders).
 2. `update_weights_chunk(update_info)` — called repeatedly. Unpacks the SkyRL packed CUDA-IPC payload, slices the contiguous buffer per param, calls `model.load_weights(weights=...)` under `set_current_vllm_config`.
 3. `finish_weight_update()` — runs `finalize_layerwise_reload` (quantization repacking, attention weight postprocessing).
@@ -48,31 +41,29 @@ vLLM worker-extension classes (loaded via `--worker-extension-cls`):
 ## Tests
 
 ```bash
-# CPU — chunk packing, transfer strategy unit tests, remote loader
+# CPU — chunk packing, transfer strategy unit tests
 uv run --extra dev pytest tests/backends/skyrl_train/weight_sync/ -v
 
-# GPU — end-to-end WorkerWrap.load_weights (NCCL + CUDA IPC paths, TP=1 and TP=2)
+# GPU — end-to-end weight sync (NCCL + CUDA IPC paths, TP=1 and TP=2)
 uv run --isolated --extra dev --extra fsdp \
   pytest tests/backends/skyrl_train/gpu/gpu_ci/inference_servers/test_weight_sync.py -v
 ```
 
-The CPU tests do **not** import `WorkerWrap` or `NewInferenceWorkerWrap`. Any change to the worker-extension classes must be exercised by the GPU test above.
+The CPU tests do **not** import `NewInferenceWorkerWrap`. Any change to the worker-extension class must be exercised by the GPU test above.
 
 ## When to touch what
 
 | Change | Run |
 |--------|-----|
 | `WeightChunk` packing / size accounting | `tests/backends/skyrl_train/weight_sync/test_weight_chunk.py` |
-| Broadcast or CUDA IPC sender/receiver | `test_transfer_strategies.py` (CPU) **and** GPU `test_weight_sync.py` |
-| `WorkerWrap` / `NewInferenceWorkerWrap` | GPU `test_weight_sync.py` (CPU tests will not catch regressions) |
-| `RemoteWeightLoader` (HTTP control plane) | `test_remote_weight_loader.py` |
+| Broadcast or CUDA IPC sender | `test_transfer_strategies.py` (CPU) **and** GPU `test_weight_sync.py` |
+| `NewInferenceWorkerWrap` | GPU `test_weight_sync.py` (CPU tests will not catch regressions) |
 
 ## vLLM version coupling
 
-`vllm` is pinned in `pyproject.toml`. Weight-sync code paths are tightly coupled to vLLM internals (`model_runner.load_weights`, `initialize_layerwise_reload`, `SKIP_TENSORS`). When bumping the pin, re-verify the GPU weight-sync tests for both the legacy and new paths.
+`vllm` is pinned in `pyproject.toml`. Weight-sync code paths are tightly coupled to vLLM internals (`model_runner.load_weights`, `initialize_layerwise_reload`, `SKIP_TENSORS`). When bumping the pin, re-verify the GPU weight-sync tests.
 
 ## Gotchas
 
-- The legacy `vllm_worker.py` and the new `new_inference_worker_wrap.py` are both live; the new path is the default (`_SKYRL_USE_NEW_INFERENCE=1`). Fixes to weight loading often need to land in **both** files.
 - NemotronH / Mamba: vLLM's layerwise reload corrupts `conv1d.weight` via shared-storage view buffers. Workaround at the top of `new_inference_worker_wrap.py` adds `"conv_weights"` to `SKIP_TENSORS` at import time. Remove pending vLLM PR #42481 (vLLM 0.21.0).
 - After `update_weights_chunk` runs, call `torch.accelerator.synchronize()` before returning so the sender doesn't drop its packed buffer mid-copy on the next barrier.

.github/workflows/gpu_skyrl_train_old_inference.yaml

@@ -8,8 +8,7 @@ uv run examples/train/gsm8k/gsm8k_dataset.py --output_dir $HOME/data/gsm8k
 
 # Run FSDP h100 tests.
 uv run --directory . --isolated --extra dev --extra fsdp pytest -s -vvv -m h100 \
-    tests/backends/skyrl_train/gpu/gpu_ci/test_policy_local_engines_e2e.py \
-    tests/backends/skyrl_train/gpu/gpu_ci/inference_servers/test_weight_sync_moe.py
+    tests/backends/skyrl_train/gpu/gpu_ci/test_policy_local_engines_e2e.py
 
 # Run Megatron h100 tests.
 uv run --directory . --isolated --extra dev --extra megatron pytest -s -vvv -m h100 \

ci/anyscale_gpu_ci_skyrl_train_old_inference.yaml

@@ -287,8 +287,8 @@
           description: ""
           objects:
             - skyrl.train.generators.base.GeneratorInterface
-            - skyrl.backends.skyrl_train.inference_engines.base.InferenceEngineInterface
-            - skyrl.backends.skyrl_train.inference_engines.inference_engine_client.InferenceEngineClient
+            - skyrl.backends.skyrl_train.inference_servers.base.InferenceEngineInterface
+            - skyrl.backends.skyrl_train.inference_servers.remote_inference_client.RemoteInferenceClient
 
     - slug: registry
       title: Algorithm Registry

ci/gpu_ci_run_h100.sh

@@ -22,24 +22,6 @@ engine reports its stats through `ray.util.metrics`, and Ray's per-node
 metrics agent exposes them at `http://<node-ip>:<MetricsExportPort>/metrics`
 in Prometheus text format. 
 
-## Inference path support
-
-| Inference path                                  | Supported |
-| ----------------------------------------------- | --------- |
-| New inference (`_SKYRL_USE_NEW_INFERENCE=1`, default) | Yes  |
-| Old inference + `generator.async_engine=true`   | Yes       |
-| Old inference + `generator.async_engine=false`  | **No**    |
-
-The new inference path ([vllm_server_actor.py:329-339](skyrl/backends/skyrl_train/inference_servers/vllm_server_actor.py#L329-L339))
-always uses `AsyncLLMEngine` and wires the stat logger unconditionally.
-
-The legacy path supports it only when `async_engine=true`
-([vllm_engine.py:359-370](skyrl/backends/skyrl_train/inference_engines/vllm/vllm_engine.py#L359-L370)).
-The synchronous `VLLMInferenceEngine` pops the flag and emits a warning
-([vllm_engine.py:240-247](skyrl/backends/skyrl_train/inference_engines/vllm/vllm_engine.py#L240-L247)):
-vLLM's sync `LLM` class doesn't accept `stat_loggers`. Set
-`generator.async_engine=true` if you need engine metrics on the legacy path.
-
 ## Metrics logged to wandb
 
 When the flag is on, the trainer constructs a `VLLMMetricsScraper`