About Clip Ratio

[jjttyy123]

Hi! I wonder what the clip ratio is of your experiment in section 5.2. I encountered an extremely high clip ratio (about 98% to 99%) when using response_length = 7144 with Qwen3.5 series models, and I wonder if this is a normal phenomenon.

[Turingzero0]

Thanks for your question. To better understand the issue, could you provide more details about your setup?

In particular, is thinking mode enabled for the Qwen3.5 model? Also, are you using a base model, an instruct model, or a model trained from the base model with SFT?

From our observations, the clip ratio is about 0.075 for a non-thinking model such as Qwen3-1.7B-SFT, while it can be much higher for thinking models. For example, DeepSeek-R1-Distill-Qwen-1.5B has a clip ratio of around 0.6 in our experiments.

That said, a clip ratio of 98%–99% seems extremely high. I would suggest first checking the rollout traces from the student model. Feel free to share more details about your model and rollout behavior so that we can look into it further.

[jjttyy123]

Thanks for your answer.

We're using the Qwen3.5‑2B base model with thinking mode disabled, which can be observed from the validation JSON logged during training. However, even with thinking mode off, we still see very long responses when evaluating the Qwen3.5‑2B base on the AMIE24 and AMIE25 validation sets: with a 32k response length limit, the average response length on these sets exceeds 17k tokens.

We tested two configurations:

1. A Qwen3.5‑2B‑SFT checkpoint (SFT following your setup — OpenThoughts3_opd.parquet with Qwen3.5‑9B rollouts, filtered to 90k rollout samples and trained for 2 epochs).
2. The plain Qwen3.5‑2B base model.

In both cases we observe the same phenomenon. At the very beginning of training (using the Qwen3.5‑9B teacher, on OpenThoughts3_opd.parquet, with response_length=7144), the mean response length is around 5,000 tokens and the clip ratio is about 30–40%. Over roughly 50 training steps, the clip ratio gradually increases to 98-99%, at which point the average response length also approaches the maximum output length of 7,144 tokens. During the same period, the true reward starts around 0.1 early in training and then declines to 0-0.01 by step 50.

Looking forward to your reply.

[Turingzero0]

Thanks for the additional details.

For the base model, we believe this high clip ratio is expected. In our observations, the EOS token / termination behavior of the base model differs from that of the post-trained teacher model. If we directly run OPD on the base model, the clip ratio can quickly increase to nearly 1, as shown in the first figure below. After inspecting the generated responses, we also found that after OPD, the base model often finishes answering the question but still fails to stop generation properly.

For the SFT model, we have released the SFT data here: https://huggingface.co/datasets/lllyx/OpenThought3-Qwen3-4B. We used this dataset to SFT the Qwen3-1.7B base model for almost one epoch. Sorry for the confusion caused by the YAML file, where epoch = 2 is specified. At that time, we were still deciding how long the SFT stage should last, so we set it this way in order to freely choose the checkpoint we wanted to use. The detailed setting is also described in the paper.

After SFT, we used OpenThoughts3_opd.parquet from the repository for OPD. Please note that this file is intended for OPD, because we de-duplicated the prompts used for SFT and OPD. The corresponding clip ratio is shown in the second figure below.

[jjttyy123]

Thanks! We will try instruct model later.