configuration of SFT on SDAR-4B-Chat

[zaiquanyang]

hi, I met a issue when reproducing SFT on SDAR-4B-Chat on a machine with 8-H20 GPUS.

I prepare the dataset sft_openr1math_sdar.json (2k samples) according to the guidance and train the SDAR-4B-Chat by the sft_sdar.yaml.

wandb:
  entity: null
  resume: 'auto'


experiment:
    project: "sft_sdar_4b_b32" # need to be same of this file name 
    num_node: 1 # the number of machines you have


model:
    pretrained_model: "/mnt/workspace/yyzq/data/dLLM/model_zoo/SDAR-4B-Chat-b32" # absolute path of your model
    optimized_name: "optimized" # the output name for your optimized model, will be saved under sft_sdar/ckpt



# sft dataset
dataset:
    optimization_data: "sft_openr1math_sdar" # "sft_openr1math_sdar"

training:
    gradient_checkpointing_enable: True # if the sequence is very larger, set as True
    gradient_accumulation_steps: 16
    batch_size_lm: 1
    mixed_precision: "bf16"
    enable_tf32: True
    seed: 10086
    num_train_epochs: 1
    max_grad_norm: 1
    method: "semi-ar" # "semi-ar""trace"
    lower_p: 0.1
    upper_p: 0.9
    block_size: 32 # the block size of your model
    shrink: 1
    post_num: 0 # number of pad token need to be trained for each data point
    max_gen_length: 2000
    max_prompt_len: 784



optimizer:
    name: adamw
    params: # default adamw params
        learning_rate: 1e-5
        scale_lr: False # scale learning rate by total batch size
        beta1: 0.9
        beta2: 0.999
        weight_decay: 0.0
        epsilon: 1e-8

lr_scheduler:
    scheduler: "cosine"
    params:
        learning_rate: ${optimizer.params.learning_rate}
        warmup_steps: 0
        min_lr_scale: 1.0

After training, the saved model achieved acc: 0.572 avg length: 619.924 while the base model is acc: 0.606 avg length: 642.392 .

The sdar_eval yaml is

experiment:
    project: "sdar_eval" # need to be same of this file name
    num_node: 1 # the number of machines you have
    node_index: 0 # no need to change



# model: "/mnt/workspace/yyzq/data/dLLM/model_zoo/SDAR-4B-Chat-b32"
model: "/mnt/workspace/yyzq/data/dLLM/dLLM-RL/sft_sdar_4b_b32/ckpt/optimized" # absolute path of your model
model_base: "sdar" # set sdar for TraDo and SDAR


# dataset you want to eval on, you need to download first, you can also modify your own dataset, see instructions in ./data
dataset:
    eval_dataset: "MATH500" #"MBPP""MATH500""GSM8K""AIME2024""GPQA""LiveCodeBench""HumanEval""LiveBench"
    data_type: "math" #"code""math"

execute:
    num_chunk: 128

rollout:
    tensor_parallel_size: 1 # set to 1 by default, if oom, try reduce max_active first, if still oom, set tensor_parallel_size to 8
    max_active: 256
    num_response_per_task: 1
    temperature: 1.0
    max_token: 4096 # max generation token num
    block_size: 32
    denoising_steps_per_block: 32
    top_p: 1.0
    top_k: 0
    remasking_strategy: "low_confidence_dynamic" #"low_confidence_static""low_confidence_dynamic"
    dynamic_threshold: 0.99
    start_with_think: False # if not reasoning model, set to False, otherwise True
    output_unmasking_history: True

There exists a large gap with the results in Table 1. I am not sure what has caused this problem.

[yinjjiew]

Are you using the block size 32 model? SDAR-4B-Chat-b32. I will check what's happening. In paper, we used block size 4

[zaiquanyang]

Yes. But I also try on SDAR-4B-Chat-b4 model and obtain the similar result as SDAR-4B-Chat-b32. I have recloned the code and sft the SDAR-4B-Chat-b4 again. The base model is acc: 0.672 avg length: 555.942 while the sft model is acc: 0.6 avg length: 608.296. The results are quite strange.

Could you reproduce the results on the SDAR-4B-Chat-b4 or SDAR-4B-Chat-b32 models?

[zaiquanyang]

Are you using the block size 32 model? SDAR-4B-Chat-b32. I will check what's happening. In paper, we used block size 4

By checking the responses after sft SDAR-4B-Chat-b4 model, I found two issues.

• The optimized model generates more long response in which the answer can not be extracted. With the more training epochs, the issue seems to have become even more serious.
• The generate response (full_output) always inlcudes several <|im_end|> tokens in the end of response like below.

-  {
    "question": "Convert the point $(0,3)$ in rectangular coordinates to polar coordinates.  Enter your answer in the form $(r,\\theta),$ where $r > 0$ and $0 \\le \\theta < 2 \\pi.$",
    "ground_truth_answer": "\\left( 3, \\frac{\\pi}{2} \\right)",
    "full_output": [
      "1. **Identify the polar coordinates:**\n   - In polar coordinates, a point is represented by $(r, \\theta)$, where $r$ is the distance from the origin to the point, and $\\theta$ is the angle from the positive $x$-axis to the line connecting the origin to the point.\n\n2. **Calculate $r$:**\n   - For the point $(0, 3)$, the distance $r$ from the origin to the point is given by the formula:\n     \\[\n     r = \\sqrt{0^2 + 3^2}\n     \\]\n   - Simplify the expression:\n     \\[\n     r = \\sqrt{9} = 3\n     \\]\n\n3. **Calculate $\\theta$:**\n   - The point $(0, 3)$ lies on the positive $y$-axis.\n   - The angle $\\theta$ from the positive $x$-axis to the line connecting the origin to the point is $\\frac{\\pi}{2}$ radians (90 degrees).\n\n4. **Combine the results:**\n   - Therefore, the polar coordinates of the point $(0, 3)$ are:\n     \\[\n     (r, \\theta) = \\left(3, \\frac{\\pi}{2}\\right)\n     \\]\n\nThe final answer is $\\boxed{\\left(3, \\frac{\\pi}{2}\\right)}$<|im_end|><|im_end|><|im_end|><|im_end|><|im_end|><|endoftext|>"
    ],
    "step_map": [],
    "extracted_output": [
      "\\left(3, \\frac{\\pi}{2}\\right)"
    ],
    "response_length": [
      296
    ],
    "prompt": "<|im_start|>user\nConvert the point $(0,3)$ in rectangular coordinates to polar coordinates.  Enter your answer in the form $(r,\\theta),$ where $r > 0$ and $0 \\le \\theta < 2 \\pi.$\nPlease reason step by step, and put your final answer within \\boxed{}.<|im_end|>\n<|im_start|>assistant",
    "correctness": [
      true
    ]
  },

But I am not sure what caused these problems...

[gdzqy26]

I also met the same issue when reproducing the sft on SDAR-chat-4b. Do you have solved that @zaiquanyang @yinjjiew?

[yinjjiew]

This config is indeed problematic. Seems that training on one node will have this issue.

For "<|im_end|><|im_end|><|im_end|><|im_end|><|im_end|><|endoftext|>", it can be solved by increase post_num to 8. But the acc is still dropping. I am looking into it.

[zaiquanyang]

@yinjjiew , do you have reproduced the result on one node ?

[yinjjiew]

Hi, sorry about the delay. To mimic 4 nodes, I tried to make lr much smaller: 1e-6 on one node (gradient accumulate step 32), the post-trained acc is 68.1 for dynamic sampling. I will try similar configs to see if the results can get better.

[zaiquanyang]

Hi, sorry about the delay. To mimic 4 nodes, I tried to make lr much smaller: 1e-6 on one node (gradient accumulate step 32), the post-trained acc is 68.1 for dynamic sampling. I will try similar configs to see if the results can get better.

Do you have some updates about this issue? I'm sorry to bother you, but it’s been a month.....

[airevo2]

I also meet this issue, reproduce with your repo on SDAR-4B-Chat, base model is 0.6713 and after trace-sft is about 0.55