Request for W&B/logs for RL reproduction

[slchenchn]

Hi, thanks for releasing the code.

I am trying to reproduce the DR.KERNEL-14B RL training using drkernel/kernel/scripts/rl/14b_coldstart_trloo_mrs_pr_prs.sh. However, my training dynamics look different from those reported in the paper.

In my run, entropy keeps decreasing, and the accuracy stays below the reported result across 170 steps.

• actor/avg_entropy: about 0.45 -> 0.13, while the paper reports relatively stable entropy around 0.5
• best last-turn Fast@1.2: about 0.20, while the paper reports 0.25

Would it be possible to share a successful W&B run, training log, or a partial/sanitized RL log?

[VPeterV]

Hi, thanks for your interest and for the question. Our experiments were conducted in the environment we used prior to the refactored public release of KernelGYM. Based on our experience, the optimal hyperparameters can vary across environments, and they may also differ between models.

For your case, I would recommend tuning hyperparameters such as COVERAGE_REWARD_WEIGHT. For example, you could try reducing it to a smaller value, such as 0.4. And here is our run report for the 14B model. You might check:

https://api.wandb.ai/links/peterv/b2yjg2a3

[slchenchn]

Hi, thanks for sharing the results.

I noticed that in the provided W&B run, the entropy appears to decrease steadily during training:

However, in Figure 5 of the paper, the entropy seems to remain relatively stable:

Could you help clarify this discrepancy?

[VPeterV]

The plot presented in the paper is based on Dr Kernel 8B, whereas Dr Kernel 14B exhibits some distinct training dynamics. Overall, we observe that larger models demonstrate improved stability; therefore, we recommend utilizing larger or more capable models to fully unlock the potential of RL.

[slchenchn]

Hi, could you also share the W&B logs of the 8B model?

[WangHaoZhe]

Hi everyone,
I'm encountering an error when executing the 8b_trloo_mrs_pr_prs.sh script. The full error message is:
AssertionError: Only free cache engine is supported for now.

Upon inspecting the script, I noticed the following variable assignment:
FREE_CACHE_ENGINE=False

Could you advise on how to resolve this conflict? Should I modify the script to FREE_CACHE_ENGINE=True?

Thanks for your help.

[slchenchn]

Hi everyone, I'm encountering an error when executing the 8b_trloo_mrs_pr_prs.sh script. The full error message is: AssertionError: Only free cache engine is supported for now.

Upon inspecting the script, I noticed the following variable assignment: FREE_CACHE_ENGINE=False

Could you advise on how to resolve this conflict? Should I modify the script to FREE_CACHE_ENGINE=True?

Thanks for your help.

I set FREE_CACHE_ENGINE=true

[WangHaoZhe]

Hi everyone, I'm encountering an error when executing the 8b_trloo_mrs_pr_prs.sh script. The full error message is: AssertionError: Only free cache engine is supported for now.
Upon inspecting the script, I noticed the following variable assignment: FREE_CACHE_ENGINE=False
Could you advise on how to resolve this conflict? Should I modify the script to FREE_CACHE_ENGINE=True?
Thanks for your help.

I set FREE_CACHE_ENGINE=true

It works. Thanks!