Skip to content

Commit 700901d

Browse files
YangZhou08YangZhou08
committed
runing
1 parent 0b714e5 commit 700901d

6 files changed

Lines changed: 556 additions & 4 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -82,10 +82,9 @@ For installation, you can simply run the following
8282
```
8383
pip install -e .[vllm]
8484
```
85-
Example script you can use to start Jackpot is here
86-
```
87-
<<placeholder>>
88-
```
85+
We prepare detailed example running scripts with Jackpot support under the following path [examples/jackpot_examples_gsm8k](examples/jackpot_examples_gsm8k) , including instructions
86+
[examples/jackpot_examples_gsm8k/README.md](examples/jackpot_examples_gsm8k/README.md)
87+
8988
Here is the detailed explanation of all the parameters we added to verl arguments for Jackpot
9089
```yaml
9190
# inside actor.yaml

‎examples/jackpot_examples_gsm8k/README.md‎

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# Jackpot GSM8K Release Examples
2+
3+
This folder provides four release-style scripts on GSM8K:
4+
5+
1. `run_qwen2-0.5b-dual-kl-gsm8k.sh`
6+
2. `run_qwen2-0.5b-dual-kl-dapo-gsm8k.sh`
7+
3. `run_qwen2-0.5b-jackpot-gsm8k.sh`
8+
4. `run_qwen3-0.6b-base-jackpot-dapo-gsm8k.sh`
9+
10+
The first two are dual-model joint training.
11+
The last two are single-model Jackpot baselines.
12+
13+
## 1) Prepare GSM8K data
14+
15+
Reference: `docs/examples/gsm8k_example.rst`
16+
17+
From repository root:
18+
19+
```bash
20+
cd examples/data_preprocess
21+
python3 gsm8k.py --local_save_dir ~/data/gsm8k
22+
```
23+
24+
Expected files:
25+
26+
- `~/data/gsm8k/train.parquet`
27+
- `~/data/gsm8k/test.parquet`
28+
29+
Each script checks these files and exits with a hint if missing.
30+
31+
## 2) Run examples
32+
33+
From repository root:
34+
35+
```bash
36+
bash examples/jackpot_gsm8k_release/run_qwen2-0.5b-dual-kl-gsm8k.sh
37+
bash examples/jackpot_gsm8k_release/run_qwen2-0.5b-dual-kl-dapo-gsm8k.sh
38+
bash examples/jackpot_gsm8k_release/run_qwen2-0.5b-jackpot-gsm8k.sh
39+
bash examples/jackpot_gsm8k_release/run_qwen3-0.6b-base-jackpot-dapo-gsm8k.sh
40+
```
41+
42+
All scripts pass extra CLI overrides through `"$@"`, so you can do:
43+
44+
```bash
45+
bash examples/jackpot_gsm8k_release/run_qwen2-0.5b-jackpot-gsm8k.sh \
46+
trainer.logger=[console] \
47+
trainer.total_epochs=2 \
48+
actor_rollout_ref.actor.optim.lr=5e-7
49+
```
50+
51+
## 3) What Jackpot arguments mean
52+
53+
These are the key Jackpot-related fields used in all scripts:
54+
55+
- `actor_rollout_ref.actor.use_jackpot`
56+
Turns Jackpot correction on or off.
57+
58+
- `actor_rollout_ref.actor.jackpot_use_latest_logits`
59+
Uses current policy logits when computing Jackpot overlap/weights.
60+
`True` usually gives tighter alignment to the model actually being updated.
61+
62+
- `actor_rollout_ref.actor.jackpot_log_probs_to_keep`
63+
Top-k width used by Jackpot overlap approximation.
64+
Larger k gives better overlap approximation but higher memory/compute.
65+
66+
- `actor_rollout_ref.actor.jackpot_lambda`
67+
Acceptance-ratio scaling factor in Jackpot correction.
68+
Increasing it makes correction stricter (fewer accepted/carrying tokens).
69+
70+
- `actor_rollout_ref.actor.jackpot_clip_ratio`
71+
Upper cap on Jackpot importance weights for stability.
72+
Lower cap is more conservative; higher cap is less biased but can be noisier.
73+
74+
- `actor_rollout_ref.actor.jackpot_use_topk_renorm`
75+
Renormalizes overlap mass in top-k space.
76+
Keep this `True` in most runs unless you intentionally study this ablation.
77+
78+
Jackpot also depends on rollout-side log-prob collection:
79+
80+
- `actor_rollout_ref.rollout.calculate_log_probs=True`
81+
- `actor_rollout_ref.rollout.log_probs_to_keep=<same top-k as actor>`
82+
83+
If these rollout settings are disabled or mismatched, Jackpot correction is not properly supported.
84+
85+
## 4) Script roles
86+
87+
- `run_qwen2-0.5b-dual-kl-gsm8k.sh`
88+
GRPO trainer, dual namespace (`large` + `small`), pairwise KL coupling.
89+
90+
- `run_qwen2-0.5b-dual-kl-dapo-gsm8k.sh`
91+
DAPO trainer variant of the same dual-namespace joint setup.
92+
93+
- `run_qwen2-0.5b-jackpot-gsm8k.sh`
94+
Single-model GRPO baseline with Jackpot only.
95+
96+
- `run_qwen3-0.6b-base-jackpot-dapo-gsm8k.sh`
97+
Single-model DAPO baseline with Jackpot only.

‎examples/jackpot_examples_gsm8k/run_qwen2-0.5b-dual-kl-dapo-gsm8k.sh‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,129 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
set -x
4+
5+
# Joint training recipe (DAPO + Jackpot) on GSM8K.
6+
# - Two namespaces (`large`, `small`) are trained jointly.
7+
# - Rollout/logprob provider is `small`.
8+
# - Pairwise KL in `trainer.topologies` couples updates between both models.
9+
10+
# Dataset preparation (from docs/examples/gsm8k_example.rst):
11+
# cd examples/data_preprocess
12+
# python3 gsm8k.py --local_save_dir ~/data/gsm8k
13+
DATA_ROOT=${DATA_ROOT:-$HOME/data/gsm8k}
14+
TRAIN_FILE=${TRAIN_FILE:-$DATA_ROOT/train.parquet}
15+
VAL_FILE=${VAL_FILE:-$DATA_ROOT/test.parquet}
16+
17+
if [[ ! -f "$TRAIN_FILE" || ! -f "$VAL_FILE" ]]; then
18+
echo "Missing GSM8K parquet files under $DATA_ROOT."
19+
echo "Run: cd examples/data_preprocess && python3 gsm8k.py --local_save_dir $DATA_ROOT"
20+
exit 1
21+
fi
22+
23+
LARGE_MODEL=${LARGE_MODEL:-Qwen/Qwen3-0.6B-Base}
24+
SMALL_MODEL=${SMALL_MODEL:-Qwen/Qwen2.5-0.5B}
25+
26+
TRAIN_BATCH_SIZE=${TRAIN_BATCH_SIZE:-64}
27+
PPO_MINI_BATCH_SIZE=${PPO_MINI_BATCH_SIZE:-64}
28+
ROLLOUT_N=${ROLLOUT_N:-8}
29+
30+
# DAPO clipping / filtering knobs.
31+
CLIP_RATIO_LOW=${CLIP_RATIO_LOW:-0.2}
32+
CLIP_RATIO_HIGH=${CLIP_RATIO_HIGH:-0.28}
33+
ENABLE_FILTER_GROUPS=${ENABLE_FILTER_GROUPS:-True}
34+
FILTER_GROUPS_METRIC=${FILTER_GROUPS_METRIC:-acc}
35+
MAX_NUM_GEN_BATCHES=${MAX_NUM_GEN_BATCHES:-10}
36+
37+
# DAPO overlong buffer knobs.
38+
ENABLE_OVERLONG_BUFFER=${ENABLE_OVERLONG_BUFFER:-True}
39+
OVERLONG_BUFFER_LEN=${OVERLONG_BUFFER_LEN:-2048}
40+
OVERLONG_PENALTY_FACTOR=${OVERLONG_PENALTY_FACTOR:-1.0}
41+
42+
# Jackpot knobs (OBRS correction):
43+
# - actor.use_jackpot=True enables Jackpot token reweighting.
44+
# - actor.jackpot_use_latest_logits=True recomputes overlap using current policy logits.
45+
# - actor.jackpot_log_probs_to_keep controls top-k width used for overlap mass estimation.
46+
# - actor.jackpot_lambda controls acceptance strictness (higher => stricter correction).
47+
# - actor.jackpot_clip_ratio caps Jackpot importance weights for stability.
48+
# - actor.jackpot_use_topk_renorm=True renormalizes overlap mass inside kept top-k.
49+
# - rollout.calculate_log_probs=True and rollout.log_probs_to_keep must stay enabled for Jackpot.
50+
JACKPOT_LOGPROBS_TO_KEEP=${JACKPOT_LOGPROBS_TO_KEEP:-20}
51+
JACKPOT_LAMBDA=${JACKPOT_LAMBDA:-1.0}
52+
JACKPOT_CLIP_RATIO=${JACKPOT_CLIP_RATIO:-16.0}
53+
54+
FWD_KL_SMALL=${FWD_KL_SMALL:-0.1}
55+
REV_KL_LARGE=${REV_KL_LARGE:-0.00}
56+
57+
python3 -m recipe.dapo.main_dapo \
58+
data.train_files="$TRAIN_FILE" \
59+
data.val_files="$VAL_FILE" \
60+
data.train_batch_size="${TRAIN_BATCH_SIZE}" \
61+
data.max_prompt_length=1024 \
62+
data.max_response_length=4096 \
63+
data.filter_overlong_prompts=True \
64+
data.truncation=error \
65+
+ray_kwargs.ray_init.object_store_memory=144000000000 \
66+
trainer.namespace=large \
67+
trainer.train_namespaces=[large,small] \
68+
trainer.rollout_from=small \
69+
trainer.critic_warmup=0 \
70+
trainer.logger=[console,wandb] \
71+
trainer.project_name=jackpot_gsm8k_release_dapo \
72+
trainer.experiment_name=qwen_dual_kl_dapo_gsm8k \
73+
trainer.n_gpus_per_node=1 \
74+
trainer.nnodes=1 \
75+
trainer.save_freq=16 \
76+
trainer.test_freq=16 \
77+
trainer.total_epochs=8 \
78+
trainer.max_actor_ckpt_to_keep=1 \
79+
trainer.val_before_train=False \
80+
trainer.validation_use_train_namespace=True \
81+
trainer.resource_pool_name=global_pool \
82+
actor_rollout_ref.model.path="${LARGE_MODEL}" \
83+
actor_rollout_ref.actor.optim.lr=1e-6 \
84+
actor_rollout_ref.model.use_remove_padding=True \
85+
actor_rollout_ref.model.enable_gradient_checkpointing=True \
86+
actor_rollout_ref.actor.ppo_mini_batch_size="${PPO_MINI_BATCH_SIZE}" \
87+
actor_rollout_ref.actor.use_dynamic_bsz=True \
88+
actor_rollout_ref.actor.ppo_max_token_len_per_gpu=9000 \
89+
actor_rollout_ref.actor.use_kl_loss=False \
90+
actor_rollout_ref.actor.clip_ratio_low="${CLIP_RATIO_LOW}" \
91+
actor_rollout_ref.actor.clip_ratio_high="${CLIP_RATIO_HIGH}" \
92+
actor_rollout_ref.actor.clip_ratio_c=10.0 \
93+
actor_rollout_ref.actor.use_jackpot=True \
94+
actor_rollout_ref.actor.jackpot_use_latest_logits=True \
95+
actor_rollout_ref.actor.jackpot_log_probs_to_keep="${JACKPOT_LOGPROBS_TO_KEEP}" \
96+
actor_rollout_ref.actor.jackpot_lambda="${JACKPOT_LAMBDA}" \
97+
actor_rollout_ref.actor.jackpot_clip_ratio="${JACKPOT_CLIP_RATIO}" \
98+
actor_rollout_ref.actor.jackpot_use_topk_renorm=True \
99+
actor_rollout_ref.actor.entropy_coeff=0 \
100+
actor_rollout_ref.actor.fsdp_config.param_offload=True \
101+
actor_rollout_ref.actor.fsdp_config.strategy=fsdp2 \
102+
actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
103+
actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
104+
actor_rollout_ref.rollout.name=vllm \
105+
actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
106+
actor_rollout_ref.rollout.n="${ROLLOUT_N}" \
107+
actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=30000 \
108+
actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=True \
109+
actor_rollout_ref.rollout.free_cache_engine=True \
110+
actor_rollout_ref.rollout.mode=sync \
111+
actor_rollout_ref.rollout.calculate_log_probs=True \
112+
actor_rollout_ref.rollout.log_probs_to_keep="${JACKPOT_LOGPROBS_TO_KEEP}" \
113+
actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=30000 \
114+
actor_rollout_ref.ref.log_prob_use_dynamic_bsz=True \
115+
actor_rollout_ref.ref.fsdp_config.param_offload=True \
116+
actor_rollout_ref.ref.fsdp_config.strategy=fsdp2 \
117+
algorithm.adv_estimator=grpo \
118+
algorithm.use_kl_in_reward=False \
119+
algorithm.filter_groups.enable="${ENABLE_FILTER_GROUPS}" \
120+
algorithm.filter_groups.max_num_gen_batches="${MAX_NUM_GEN_BATCHES}" \
121+
algorithm.filter_groups.metric="${FILTER_GROUPS_METRIC}" \
122+
reward_model.reward_manager=dapo \
123+
reward_model.overlong_buffer.enable="${ENABLE_OVERLONG_BUFFER}" \
124+
reward_model.overlong_buffer.len="${OVERLONG_BUFFER_LEN}" \
125+
reward_model.overlong_buffer.penalty_factor="${OVERLONG_PENALTY_FACTOR}" \
126+
"+trainer.topologies=[{name:dual_kl,rollout:small,logprob:small,train:[small,large],logprob_map:{large:large},kl_pairs:[{name:fwd_small_vs_large,train:small,p:large,q:small,mode:forward,coef:${FWD_KL_SMALL}},{name:rev_large_vs_small,train:large,p:large,q:small,mode:reverse,coef:${REV_KL_LARGE},use_is:true}]}]" \
127+
'+trainer.topology_loop=[{name:dual_kl,repeat:1}]' \
128+
"+trainer.worker_namespaces=[{name:small,train:true,config:{trainer:{rollout_from:small},actor_rollout_ref:{model:{path:'${SMALL_MODEL}'},actor:{optim:{lr:1e-6},ppo_mini_batch_size:${PPO_MINI_BATCH_SIZE},use_dynamic_bsz:true,ppo_max_token_len_per_gpu:12000,use_jackpot:true,jackpot_use_latest_logits:true,jackpot_log_probs_to_keep:${JACKPOT_LOGPROBS_TO_KEEP},jackpot_lambda:${JACKPOT_LAMBDA},jackpot_clip_ratio:${JACKPOT_CLIP_RATIO},jackpot_use_topk_renorm:true,fsdp_config:{param_offload:true,optimizer_offload:true}},ref:{log_prob_max_token_len_per_gpu:30000},rollout:{tensor_model_parallel_size:1,gpu_memory_utilization:0.6,log_prob_max_token_len_per_gpu:30000,log_prob_use_dynamic_bsz:true,free_cache_engine:true,mode:sync,calculate_log_probs:true,log_probs_to_keep:${JACKPOT_LOGPROBS_TO_KEEP}}}}}]" \
129+
"$@"

‎examples/jackpot_examples_gsm8k/run_qwen2-0.5b-dual-kl-gsm8k.sh‎

Lines changed: 113 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,113 @@
1+
#!/usr/bin/env bash
2+
set -euo pipefail
3+
set -x
4+
5+
# Joint training recipe (GRPO + Jackpot) on GSM8K.
6+
# - Two namespaces are updated together: `large` and `small`.
7+
# - Rollout/logprob generation comes from `small` to reduce serving cost.
8+
# - Pairwise KL terms in `trainer.topologies` couple both models during updates.
9+
10+
# Dataset preparation (from docs/examples/gsm8k_example.rst):
11+
# cd examples/data_preprocess
12+
# python3 gsm8k.py --local_save_dir ~/data/gsm8k
13+
DATA_ROOT=${DATA_ROOT:-$HOME/data/gsm8k}
14+
TRAIN_FILE=${TRAIN_FILE:-$DATA_ROOT/train.parquet}
15+
VAL_FILE=${VAL_FILE:-$DATA_ROOT/test.parquet}
16+
17+
if [[ ! -f "$TRAIN_FILE" || ! -f "$VAL_FILE" ]]; then
18+
echo "Missing GSM8K parquet files under $DATA_ROOT."
19+
echo "Run: cd examples/data_preprocess && python3 gsm8k.py --local_save_dir $DATA_ROOT"
20+
exit 1
21+
fi
22+
23+
# Large model is the main namespace; small model provides rollout/logprob service.
24+
LARGE_MODEL=${LARGE_MODEL:-Qwen/Qwen3-0.6B-Base}
25+
SMALL_MODEL=${SMALL_MODEL:-Qwen/Qwen2.5-0.5B}
26+
27+
# Throughput-related knobs.
28+
TRAIN_BATCH_SIZE=${TRAIN_BATCH_SIZE:-128}
29+
PPO_MINI_BATCH_SIZE=${PPO_MINI_BATCH_SIZE:-32}
30+
ROLLOUT_N=${ROLLOUT_N:-8}
31+
32+
# Jackpot knobs (OBRS correction):
33+
# - actor.use_jackpot=True enables Jackpot token reweighting.
34+
# - actor.jackpot_use_latest_logits=True recomputes overlap using current policy logits.
35+
# - actor.jackpot_log_probs_to_keep controls top-k width used for overlap mass estimation.
36+
# - actor.jackpot_lambda controls acceptance strictness (higher => stricter correction).
37+
# - actor.jackpot_clip_ratio caps Jackpot importance weights for stability.
38+
# - actor.jackpot_use_topk_renorm=True renormalizes overlap mass inside kept top-k.
39+
# - rollout.calculate_log_probs=True and rollout.log_probs_to_keep must stay enabled for Jackpot.
40+
JACKPOT_LOGPROBS_TO_KEEP=${JACKPOT_LOGPROBS_TO_KEEP:-20}
41+
JACKPOT_LAMBDA=${JACKPOT_LAMBDA:-1.0}
42+
JACKPOT_CLIP_RATIO=${JACKPOT_CLIP_RATIO:-16.0}
43+
44+
# KL coupling knobs for joint training.
45+
# - FWD_KL_SMALL: forward KL used when training `small`.
46+
# - REV_KL_LARGE: reverse KL used when training `large`.
47+
FWD_KL_SMALL=${FWD_KL_SMALL:-0.1}
48+
REV_KL_LARGE=${REV_KL_LARGE:-0.02}
49+
50+
python3 -m verl.trainer.main_ppo \
51+
data.train_files="$TRAIN_FILE" \
52+
data.val_files="$VAL_FILE" \
53+
data.train_batch_size="${TRAIN_BATCH_SIZE}" \
54+
data.max_prompt_length=1024 \
55+
data.max_response_length=4096 \
56+
data.val_max_samples=1000 \
57+
data.filter_overlong_prompts=True \
58+
data.truncation=error \
59+
+ray_kwargs.ray_init.object_store_memory=144000000000 \
60+
trainer.namespace=large \
61+
trainer.train_namespaces=[large,small] \
62+
trainer.rollout_from=small \
63+
trainer.critic_warmup=0 \
64+
trainer.logger=[console,wandb] \
65+
trainer.project_name=jackpot_gsm8k_release \
66+
trainer.experiment_name=qwen_dual_kl_gsm8k \
67+
trainer.n_gpus_per_node=1 \
68+
trainer.nnodes=1 \
69+
trainer.save_freq=8 \
70+
trainer.test_freq=8 \
71+
trainer.total_epochs=8 \
72+
trainer.max_actor_ckpt_to_keep=1 \
73+
trainer.val_before_train=True \
74+
trainer.validation_use_train_namespace=True \
75+
trainer.resource_pool_name=global_pool \
76+
actor_rollout_ref.model.path="${LARGE_MODEL}" \
77+
actor_rollout_ref.actor.optim.lr=1e-6 \
78+
actor_rollout_ref.model.use_remove_padding=True \
79+
actor_rollout_ref.model.enable_gradient_checkpointing=True \
80+
actor_rollout_ref.actor.ppo_mini_batch_size="${PPO_MINI_BATCH_SIZE}" \
81+
actor_rollout_ref.actor.use_dynamic_bsz=True \
82+
actor_rollout_ref.actor.ppo_max_token_len_per_gpu=7000 \
83+
actor_rollout_ref.actor.use_kl_loss=False \
84+
actor_rollout_ref.actor.entropy_coeff=0 \
85+
actor_rollout_ref.actor.fsdp_config.param_offload=True \
86+
actor_rollout_ref.actor.fsdp_config.strategy=fsdp2 \
87+
actor_rollout_ref.actor.fsdp_config.optimizer_offload=True \
88+
actor_rollout_ref.actor.use_jackpot=True \
89+
actor_rollout_ref.actor.jackpot_use_latest_logits=True \
90+
actor_rollout_ref.actor.jackpot_log_probs_to_keep="${JACKPOT_LOGPROBS_TO_KEEP}" \
91+
actor_rollout_ref.actor.jackpot_lambda="${JACKPOT_LAMBDA}" \
92+
actor_rollout_ref.actor.jackpot_clip_ratio="${JACKPOT_CLIP_RATIO}" \
93+
actor_rollout_ref.actor.jackpot_use_topk_renorm=True \
94+
actor_rollout_ref.rollout.tensor_model_parallel_size=1 \
95+
actor_rollout_ref.rollout.name=vllm \
96+
actor_rollout_ref.rollout.gpu_memory_utilization=0.6 \
97+
actor_rollout_ref.rollout.n="${ROLLOUT_N}" \
98+
actor_rollout_ref.rollout.log_prob_max_token_len_per_gpu=30000 \
99+
actor_rollout_ref.rollout.log_prob_use_dynamic_bsz=True \
100+
actor_rollout_ref.rollout.free_cache_engine=True \
101+
actor_rollout_ref.rollout.mode=sync \
102+
actor_rollout_ref.rollout.calculate_log_probs=True \
103+
actor_rollout_ref.rollout.log_probs_to_keep="${JACKPOT_LOGPROBS_TO_KEEP}" \
104+
actor_rollout_ref.ref.log_prob_max_token_len_per_gpu=30000 \
105+
actor_rollout_ref.ref.log_prob_use_dynamic_bsz=True \
106+
actor_rollout_ref.ref.fsdp_config.param_offload=True \
107+
actor_rollout_ref.ref.fsdp_config.strategy=fsdp2 \
108+
algorithm.adv_estimator=grpo \
109+
algorithm.use_kl_in_reward=False \
110+
"+trainer.topologies=[{name:dual_kl,rollout:small,logprob:small,train:[small,large],logprob_map:{large:large},kl_pairs:[{name:fwd_small_vs_large,train:small,p:large,q:small,mode:forward,coef:${FWD_KL_SMALL}},{name:rev_large_vs_small,train:large,p:large,q:small,mode:reverse,coef:${REV_KL_LARGE},use_is:true}]}]" \
111+
'+trainer.topology_loop=[{name:dual_kl,repeat:1}]' \
112+
"+trainer.worker_namespaces=[{name:small,train:true,config:{trainer:{rollout_from:small},actor_rollout_ref:{model:{path:'${SMALL_MODEL}'},actor:{optim:{lr:1e-6},ppo_mini_batch_size:${PPO_MINI_BATCH_SIZE},use_dynamic_bsz:true,ppo_max_token_len_per_gpu:12000,use_jackpot:true,jackpot_use_latest_logits:true,jackpot_log_probs_to_keep:${JACKPOT_LOGPROBS_TO_KEEP},jackpot_lambda:${JACKPOT_LAMBDA},jackpot_clip_ratio:${JACKPOT_CLIP_RATIO},jackpot_use_topk_renorm:true,fsdp_config:{param_offload:true,optimizer_offload:true}},ref:{log_prob_max_token_len_per_gpu:30000},rollout:{tensor_model_parallel_size:1,gpu_memory_utilization:0.6,log_prob_max_token_len_per_gpu:30000,log_prob_use_dynamic_bsz:true,free_cache_engine:true,mode:sync,calculate_log_probs:true,log_probs_to_keep:${JACKPOT_LOGPROBS_TO_KEEP}}}}}]" \
113+
"$@"

0 commit comments

Comments
 (0)