SFT training data

[hshc123]

Thank you for your excellent work. I’d like to ask a couple of questions:

For TraDo-8B-Thinking, is the long CoT SFT data the dataset named demon_openr1math.json that’s included in the code repository?
Will this kind of sequential “chain-of-thought” reasoning suppress the parallel generation capability of the DLM?

[yinjjiew]

No. The long-cot data we used is 75K samples from OpenThoughts3-1.2M.

Based on the generate.py in this repo, using DLM's dynmaic sampling is indeed faster than AR LLM, no matter what kinds of reasoning pattern it is. But indeed, given these reasoning tokens (wait, alternatively) frequently appeared in long-cot reasoning, it could suppress the parallel generation capability ( we can test if it is true later