feat(grpo): add RLSD self-distilled advantage reweighting (#9758)

* feat(grpo): add RLSD self-distilled advantage reweighting

Introduce RLSD (Self-Distilled RLVR) token-level advantage reweighting for
GRPO, enabled via `--advantage_reweight rlsd`. The per-sequence advantage is
redistributed inside each trajectory using the teacher-vs-student logprob gap
as a sign-aware, clipped multiplicative weight, without flipping the reward
sign. The teacher reuses the OPSD self-distillation forward (current policy
conditioned on a privileged `teacher_prompt` column), and a frozen local
teacher via `--teacher_model` is also supported.

- add `apply_rlsd_reweight` in rl_core/advantage.py
- add RLSD args and lambda warmup/decay schedule to the GRPO trainer
- add `_check_rlsd` fail-fast validation in RLHFArguments
- add unit tests in tests/train/test_rlsd_reweight.py

* ci(workflows): add Qoder code review and assistant

* docs(grpo): document RLSD advantage reweighting parameters

Add the new RLSD (Self-Distilled RLVR) CLI parameters to the English and
Chinese command-line references: advantage_reweight, rlsd_lambda,
rlsd_reweight_clip_range, rlsd_lambda_warmup_steps, rlsd_lambda_decay_steps
and rlsd_negative_only, including defaults, valid ranges, the reweight
formula, enable requirements and the two teacher modes.

* update core-review.yaml

Added separate jobs for trusted and external code reviews in GitHub Actions workflow.

---------

Co-authored-by: jinghanhu <hujinghan.hjh@alibaba-inc.com>

Author: qychen2001

.github/workflows/assistant.yaml

@@ -0,0 +1,60 @@
+name: Qoder Assistant
+
+on:
+  issue_comment:
+    types: [created]
+  pull_request_review_comment:
+    types: [created]
+
+jobs:
+  qoder-assistant:
+    if: |
+      contains(github.event.comment.body, '@qoder') &&
+      !endsWith(github.event.comment.user.login, '[bot]')
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      issues: write
+      pull-requests: write
+      id-token: write
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Build Arguments
+        id: build_args
+        run: |
+          ARGS="REPO: ${{ github.repository }}
+          REQUEST_SOURCE: ${{ github.event_name }}
+          THREAD_ID: ${{ github.event.comment.node_id }}
+          COMMENT_ID: ${{ github.event.comment.id }}
+          AUTHOR: ${{ github.event.comment.user.login }}
+          BODY: ${{ github.event.comment.body }}
+          URL: ${{ github.event.comment.html_url }}
+          IS_PR: ${{ github.event.issue.pull_request != null || github.event_name == 'pull_request_review_comment' }}
+          ISSUE_OR_PR_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number }}"
+
+          if [ -n "${{ github.event.comment.pull_request_review_id }}" ]; then
+            ARGS="$ARGS
+          REVIEW_ID: ${{ github.event.comment.pull_request_review_id }}"
+          fi
+
+          if [ -n "${{ github.event.comment.in_reply_to_id }}" ]; then
+            ARGS="$ARGS
+          REPLY_TO_COMMENT_ID: ${{ github.event.comment.in_reply_to_id }}"
+          fi
+
+          echo "args<<EOF" >> $GITHUB_OUTPUT
+          echo "$ARGS" >> $GITHUB_OUTPUT
+          echo "EOF" >> $GITHUB_OUTPUT
+
+      - name: Run Qoder Assistant
+        uses: QoderAI/qoder-action@v0
+        with:
+          qoder_personal_access_token: ${{ secrets.QODER_PERSONAL_ACCESS_TOKEN }}
+          prompt: |
+            /assistant
+            ${{ steps.build_args.outputs.args }}

.github/workflows/code-review.yaml

@@ -0,0 +1,53 @@
+name: Qoder Auto Code Review
+
+on:
+  pull_request_target:
+    types: [opened, synchronize, reopened]
+
+jobs:
+  qoder-review-trusted:
+    if: contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.pull_request.author_association)
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      pull-requests: write
+      id-token: write
+
+    steps:
+      - name: Checkout PR head
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          ref: ${{ github.event.pull_request.head.sha }}
+
+      - name: Run Qoder Code Review
+        uses: QoderAI/qoder-action@v0
+        with:
+          qoder_personal_access_token: ${{ secrets.QODER_PERSONAL_ACCESS_TOKEN }}
+          prompt: |
+            /review-pr
+            REPO:${{ github.repository }} PR_NUMBER:${{ github.event.pull_request.number }}
+
+  qoder-review-external:
+    if: ${{ !contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.pull_request.author_association) }}
+    runs-on: ubuntu-latest
+    environment: qoder-review
+    permissions:
+      contents: read
+      pull-requests: write
+      id-token: write
+
+    steps:
+      - name: Checkout PR head
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          ref: ${{ github.event.pull_request.head.sha }}
+
+      - name: Run Qoder Code Review
+        uses: QoderAI/qoder-action@v0
+        with:
+          qoder_personal_access_token: ${{ secrets.QODER_PERSONAL_ACCESS_TOKEN }}
+          prompt: |
+            /review-pr
+            REPO:${{ github.repository }} PR_NUMBER:${{ github.event.pull_request.number }}

docs/source/Instruction/Command-line-parameters.md

@@ -679,6 +679,12 @@ reward模型参数将在PPO、GRPO中使用；teacher模型参数在GKD与GRPO
   - 注意：`gdpo` 模式不支持 `kl_in_reward=True`，若同时设置会自动将 `kl_in_reward` 设为 `False`。
   - GDPO 适用于多奖励优化场景：当使用多个奖励函数时，GDPO 会对每个奖励函数分别在组内进行标准化（减均值、除标准差），然后使用 `reward_weights` 进行加权求和，最后再进行批次级别的标准化。这种方式可以更好地保留各个奖励的相对差异，避免不同奖励组合坍塌成相同的 advantage 值。
 - teacher_kl_coef: OPD-RL中teacher_kl的系数，即 `adv_t = base_adv + teacher_kl_coef * teacher_kl`。默认为 1.0。
+- advantage_reweight: 优势重加权算法，默认为None（不启用）。可选项为 `rlsd`（[RLSD, Self-Distilled RLVR](https://arxiv.org/abs/2604.03128)）：利用 teacher 与 student 在相同采样 token 上的对数概率差异，将 GRPO 的序列级优势在 token 维度重新分配，即 `delta_t = logP_T(y_t) - logP_S(y_t)`、`w_t = exp(sign(A) * delta_t)`、`reweight = (1 - λ) + λ * clip(w_t)`、`A_hat_t = A * reweight`；reweight 恒为正，不会翻转环境奖励的符号。启用后需满足：已设置 `reward_funcs`（reweight 方向依赖优势符号 `sign(A)`），且数据集包含 `teacher_prompt` 列（特权 prompt，如 问题 + 参考解答/标准答案）；与 `use_liger_kernel`、`loss_type in [real, fipo]`、`off_policy_sequence_mask_delta`、`teacher_model_server` 不兼容。teacher 模式：不设置 `--teacher_model` 时为动态自蒸馏（teacher = 当前策略在特权 prompt 上打分）；设置 `--teacher_model <ckpt>` 时为冻结本地 teacher（在 no_grad 下打分，训练中不更新）。
+- rlsd_lambda: RLSD 混合权重，`reweight = (1 - λ) + λ * clip(w_t)`。`0` 退化为普通 GRPO，`1` 为完整 RLSD 重加权。取值需在 [0, 1]，默认为 0.5。
+- rlsd_reweight_clip_range: 证据权重裁剪范围 eps_w，将 `w_t = exp(sign(A) * delta_t)` 裁剪到 `[1 - eps_w, 1 + eps_w]`，避免 teacher/student 对数概率差异造成过大的倍率。需 >= 0，默认为 0.2。
+- rlsd_lambda_warmup_steps: 将 λ 从 0 线性 warmup 到 `rlsd_lambda` 的步数。默认为 0（不进行 warmup）。
+- rlsd_lambda_decay_steps: 在 warmup 之后，将 λ 从 `rlsd_lambda` 线性衰减到 0 的步数。默认为 0（不衰减，λ 保持恒定）。
+- rlsd_negative_only: 是否仅对优势为负（`A < 0`，即错误回答）的序列进行重加权；优势非负的序列保留普通 GRPO 优势。默认为 False。
 - sync_ref_model: 是否定期同步ref_model，默认为False。
   - ref_model_mixup_alpha: 控制在更新过程中model和先前ref_model之间的混合。更新公式为 $π_{ref} = α * π_θ + (1 - α) * π_{ref_{prev}}$。默认为0.6。
   - ref_model_sync_steps: 同步频率，默认为512。

docs/source_en/Instruction/Command-line-parameters.md

@@ -696,6 +696,12 @@ The hyperparameters for the reward function can be found in the [Built-in Reward
   - Note: `gdpo` mode does not support `kl_in_reward=True`. If both are set, `kl_in_reward` will be automatically set to `False`.
   - GDPO is designed for multi-reward optimization: When using multiple reward functions, GDPO normalizes each reward function separately within groups (subtract mean, divide by std), then performs weighted aggregation using `reward_weights`, and finally applies batch-level normalization. This approach better preserves the relative differences between rewards and prevents different reward combinations from collapsing into identical advantage values.
 - teacher_kl_coef: Coefficient for teacher KL in OPD-RL, i.e. `adv_t = base_adv + teacher_kl_coef * teacher_kl`. Default is 1.0.
+- advantage_reweight: Advantage reweighting algorithm. Default is None (disabled). Option: `rlsd` ([RLSD, Self-Distilled RLVR](https://arxiv.org/abs/2604.03128)), which redistributes the per-sequence GRPO advantage across tokens using the teacher-vs-student log-probability gap on the same sampled tokens, i.e. `delta_t = logP_T(y_t) - logP_S(y_t)`, `w_t = exp(sign(A) * delta_t)`, `reweight = (1 - λ) + λ * clip(w_t)`, `A_hat_t = A * reweight`. The reweight is strictly positive and never flips the sign of the environment reward. When enabled it requires `reward_funcs` (the reweight direction uses `sign(A)`) and a `teacher_prompt` column in the dataset (the privileged prompt, e.g. question + reference solution / ground-truth answer); it is incompatible with `use_liger_kernel`, `loss_type in [real, fipo]`, `off_policy_sequence_mask_delta`, and `teacher_model_server`. Teacher modes: without `--teacher_model` it runs dynamic self-distillation (teacher = current policy scored on the privileged prompt); with `--teacher_model <ckpt>` it uses a frozen local teacher (scored under no_grad, never updated during training).
+- rlsd_lambda: RLSD mixing weight, `reweight = (1 - λ) + λ * clip(w_t)`. `0` reduces to plain GRPO, `1` is full RLSD reweighting. Must be in [0, 1]. Default is 0.5.
+- rlsd_reweight_clip_range: Evidence-weight clip range eps_w. Clips `w_t = exp(sign(A) * delta_t)` to `[1 - eps_w, 1 + eps_w]` to prevent overly large multipliers from the teacher/student log-prob gap. Must be >= 0. Default is 0.2.
+- rlsd_lambda_warmup_steps: Number of steps to linearly warm up λ from 0 to `rlsd_lambda`. Default is 0 (no warmup).
+- rlsd_lambda_decay_steps: Number of steps to linearly decay λ from `rlsd_lambda` to 0 after warmup. Default is 0 (no decay; λ stays constant).
+- rlsd_negative_only: Whether to reweight only sequences with negative advantage (`A < 0`, i.e. incorrect responses); sequences with non-negative advantage keep plain GRPO advantages. Default is False.
 - sync_ref_model: Whether to synchronize the reference model. Default is False.
   - ref_model_mixup_alpha: The Parameter controls the mix between the current policy and the previous reference policy during updates. The reference policy is updated according to the equation: $π_{ref} = α * π_θ + (1 - α) * π_{ref_{prev}}$. Default is 0.6.
   - ref_model_sync_steps：The parameter determines how frequently the current policy is synchronized with the reference policy. Default is 512.

swift/arguments/rlhf_args.py

@@ -568,6 +568,62 @@ def _check_grpo(self):
             raise NotImplementedError('Currently, async_generate is not supported with multi-turn functionality.')
 
         self._check_opd_rl()
+        self._check_rlsd()
+
+    def _check_rlsd(self):
+        """Validate RLSD (Self-Distilled RLVR) advantage reweighting parameters.
+
+        RLSD reweights the per-sequence GRPO advantage per token using the teacher-vs-student
+        logprob gap. It relies on an OPSD teacher forward driven by a per-sample ``teacher_prompt``
+        column (privileged context, e.g. question + reference solution / GT answer). It also needs
+        an environment reward (``reward_funcs``) because the reweight direction uses ``sign(A)``.
+
+        Two teacher modes are supported (mirroring the reference's ``freeze_teacher_model`` flag):
+
+        * **Dynamic self-distillation** (no ``--teacher_model``, default): teacher = current policy
+          evaluated on the privileged prompt (paper's same-model, two-contexts Δ_t; equivalent to
+          the reference's ``freeze_teacher_model=false``).
+        * **Frozen local teacher** (``--teacher_model <ckpt>``): a separate model scores the
+          privileged prompt under ``no_grad`` and is never in the optimizer, so it stays frozen
+          throughout training. Point ``--teacher_model`` at the initial policy checkpoint to
+          reproduce the reference's ``freeze_teacher_model=true`` (frozen θ₀ anchor).
+        """
+        if self.advantage_reweight != 'rlsd':
+            return
+        if not (0.0 <= self.rlsd_lambda <= 1.0):
+            raise ValueError(f'rlsd_lambda must be in [0, 1], got {self.rlsd_lambda}.')
+        if self.rlsd_reweight_clip_range < 0:
+            raise ValueError(f'rlsd_reweight_clip_range (eps_w) must be >= 0, got {self.rlsd_reweight_clip_range}.')
+        if not self.reward_funcs:
+            raise ValueError('advantage_reweight=rlsd requires reward_funcs: the reweight direction uses '
+                             'sign(A) of the GRPO advantage, which needs an environment reward.')
+        if self.use_liger_kernel:
+            raise ValueError('advantage_reweight=rlsd is not compatible with use_liger_kernel '
+                             '(the fused loss path bypasses the per-token RLSD reweight).')
+        if self.loss_type in ['real', 'fipo']:
+            raise ValueError(f'advantage_reweight=rlsd does not support loss_type={self.loss_type!r} '
+                             '(it reduces the advantage to a per-sequence scalar).')
+        if self.off_policy_sequence_mask_delta is not None:
+            raise ValueError('advantage_reweight=rlsd does not support off_policy_sequence_mask_delta.')
+        # Local frozen teacher (--teacher_model) is allowed: it drives the OPSD teacher forward on
+        # the privileged prompt via the same code path as OPD-RL, and use_rlsd bypasses OPD-RL's
+        # additive term (grpo_trainer.py:266) so there is no double-count. The API-teacher path
+        # (--teacher_model_server) scores tokens over a remote inference call whose tokenizer
+        # alignment with the student is not guaranteed, so keep it gated for RLSD's per-token reweight.
+        if self.teacher_model_server is not None:
+            raise ValueError('advantage_reweight=rlsd does not support --teacher_model_server (remote API '
+                             'teacher). Use --teacher_model <local ckpt> for a frozen local teacher, or omit '
+                             'both to use dynamic self-distillation (teacher = current policy).')
+        if self.beta not in (0, 0.0):
+            logger.warning(f'advantage_reweight=rlsd: the reference RLSD config disables KL (beta=0), '
+                           f'but beta={self.beta}. Consider setting --beta 0.')
+        if self.teacher_model is not None:
+            logger.info(f'advantage_reweight=rlsd: frozen-teacher mode — --teacher_model={self.teacher_model!r} scores '
+                        f'the `teacher_prompt` column under no_grad and stays frozen throughout training (matches the '
+                        f'RLSD reference freeze_teacher_model=true when this checkpoint is the initial policy).')
+        else:
+            logger.info('advantage_reweight=rlsd: dynamic self-distillation mode — expecting a `teacher_prompt` '
+                        'column in the dataset; teacher = current policy on the privileged prompt.')
 
     def _check_opd_rl(self):
         """Fail-fast OPD-RL (teacher distillation on GRPO) parameter compatibility.

swift/rl_core/advantage.py

@@ -279,6 +279,58 @@ def expand_advantage_to_per_token(
     return per_token_adv
 
 
+def apply_rlsd_reweight(
+    base_advantages: torch.Tensor,
+    completion_mask: torch.Tensor,
+    teacher_per_token_logps: torch.Tensor,
+    policy_per_token_logps: torch.Tensor,
+    lam: float,
+    clip_range: float,
+    negative_only: bool = False,
+) -> torch.Tensor:
+    """RLSD (Self-Distilled RLVR) token-level advantage reweighting.
+
+    Redistributes the per-sequence GRPO advantage *inside* each trajectory using the
+    teacher-vs-student log-prob gap, without ever flipping the sign of the environment
+    reward (the reweight is strictly positive). Mirrors ``_build_stgca_advantages`` in
+    the reference implementation (RLSD/verl/workers/actor/dp_opsd_actor.py)::
+
+        delta_t  = stop_grad(logP_T(y_t) - logP_S(y_t))
+        w_t      = exp(sign(A) * delta_t)
+        reweight = (1 - lam) + lam * clip(w_t, 1 - clip_range, 1 + clip_range)
+        A_hat_t  = A * stop_grad(reweight)
+
+    ``lam`` mixes between pure GRPO (``lam=0`` -> plain broadcast) and full RLSD reweighting
+    (``lam=1``). The teacher is the "informed self": the same policy conditioned on the
+    ground-truth answer, scoring the *same* sampled tokens (``teacher_per_token_logps``).
+    ``policy_per_token_logps`` is the student side (the batch-time old logps).
+
+    Args:
+        base_advantages: ``[B]`` per-sequence GRPO advantage.
+        completion_mask: ``[B, T]`` response-token mask.
+        teacher_per_token_logps: ``[B, T]`` teacher logp on sampled tokens.
+        policy_per_token_logps: ``[B, T]`` student (old) logp on the same tokens.
+        lam: Effective mixing weight (already schedule-adjusted).
+        clip_range: Evidence-weight clip epsilon ``eps_w``.
+        negative_only: When True, only reweight sequences with ``A < 0`` (incorrect
+            responses); ``A >= 0`` sequences keep pure GRPO advantages.
+
+    Returns:
+        ``[B, T]`` per-token reweighted advantage (masked outside the response).
+    """
+    mask = completion_mask
+    base_bt = base_advantages.unsqueeze(1).expand_as(mask)
+    delta = (teacher_per_token_logps.detach() - policy_per_token_logps.detach()) * mask
+    sign_a = torch.sign(base_bt)
+    weights = torch.exp(sign_a * delta) * mask
+    clipped = torch.clamp(weights, min=1.0 - clip_range, max=1.0 + clip_range)
+    reweight = (1.0 - lam) + lam * clipped
+    if negative_only:
+        seq_neg = (base_advantages < 0).float().unsqueeze(1)
+        reweight = seq_neg * reweight + (1.0 - seq_neg)
+    return base_bt * reweight.detach() * mask
+
+
 @dataclass
 class RewardMetrics:
     """Reward statistics for logging."""

swift/rlhf_trainers/args_mixin.py

@@ -436,6 +436,17 @@ class GRPOArgumentsMixin(RolloutTrainerArgumentsMixin):
     # enabled when a teacher (teacher_model / teacher_model_server) is set on a GRPO run.
     teacher_kl_coef: float = 1.0
 
+    # RLSD (Self-Distilled RLVR), https://arxiv.org/abs/2604.03128
+    # Token-level advantage reweighting using the teacher-vs-student logprob gap. Reuses the OPSD
+    # self-distillation teacher forward (teacher = current policy conditioned on the ground-truth
+    # answer via a per-sample ``teacher_prompt`` column). Set ``advantage_reweight='rlsd'`` to enable.
+    advantage_reweight: Optional[Literal['rlsd']] = None
+    rlsd_lambda: float = 0.5  # mixing weight: 0 -> pure GRPO, 1 -> full RLSD reweighting
+    rlsd_reweight_clip_range: float = 0.2  # eps_w: clip evidence weight to [1-eps_w, 1+eps_w]
+    rlsd_lambda_warmup_steps: int = 0  # linear warmup of lambda from 0 to rlsd_lambda
+    rlsd_lambda_decay_steps: int = 0  # linear decay of lambda to 0 over this many steps
+    rlsd_negative_only: bool = False  # only reweight sequences with advantage < 0
+
     # REAL https://arxiv.org/abs/2602.05630
     real_tau: float = 0.5