What does this PR do ?

Continue working on the existing CISPO PR #2187.

Adds CISPO support to the GRPO loss path and provides matched GRPO/CISPO Qwen3-30B-A3B high-off-policy recipes to validate the clipped importance-sampling objective under repeated updates per rollout.

Notes:

This PR at this moment includes the yaml and sh files that can be used to fully reproduce the results below, can remove them later.
PR with test files cleaned.

Issues

N/A

Usage

You can potentially add a usage example below.

# GRPO baseline
bash tests/test_suites/llm/cispo-mm1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-grpo.sh

# CISPO treatment
bash tests/test_suites/llm/cispo-mm1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-cispo.sh

The recipes use Qwen3-30B-A3B with Megatron policy training and colocated vLLM generation. They are matched except for the loss configuration:

• GRPO: standard clipped ratio objective.
• CISPO: detached clipped IS-weight objective with a wider upper bound.

Validation Results

High-off-policy validation was run with Qwen3-30B-A3B on 2 nodes x 8 GPUs. The setup uses 32 prompts x 16 generations = 512 trajectories per rollout and train_global_batch_size=32, giving 16 policy updates per rollout.

Mechanistic checks:

Async lag-1 high-off-policy validation was also run with the same model and rollout/update ratio, but with non-colocated async vLLM generation and max_trajectory_age_steps=1.

Async lag-1 mechanistic checks:

Summary: Across the synchronous high-off-policy and async lag-1 validations, CISPO shows modest positive evidence: average train reward is roughly tied to slightly higher, average validation accuracy is slightly higher, and the async lag-1 run finishes with higher final validation accuracy. The effect is not a clean sweep at every checkpoint, but CISPO is at least competitive with GRPO and appears more stable on the ratio/KL outlier diagnostics in the async lag-1 setting.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you run the unit tests and functional tests locally? Visit our Testing Guide for how to run tests
  • Functional validation: 100-step GRPO/CISPO high-off-policy runs on Qwen3-30B-A3B completed.
  • Functional validation: 100-step GRPO/CISPO async lag-1 high-off-policy runs on Qwen3-30B-A3B completed.
• Did you add or update any necessary documentation? Visit our Document Development Guide for how to write, build and test the docs.

Additional Information

CISPO validation used the final high-off-policy recipes:

• examples/configs/recipes/llm/cispo-mm1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-grpo.yaml
• examples/configs/recipes/llm/cispo-mm1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-cispo.yaml
• examples/configs/recipes/llm/cispo-mm1-async-lag1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-grpo.yaml
• examples/configs/recipes/llm/cispo-mm1-async-lag1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-cispo.yaml

The corresponding launch scripts are:

• tests/test_suites/llm/cispo-mm1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-grpo.sh
• tests/test_suites/llm/cispo-mm1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-cispo.sh
• tests/test_suites/llm/cispo-mm1-async-lag1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-grpo.sh
• tests/test_suites/llm/cispo-mm1-async-lag1-highoffpolicy-qwen3-30ba3b-2n8g-megatron-cispo.sh