QLoRA and DDP

[mrT23]

Thanks for the great repo

i have two questions about training the models (specifically WizardCoder):

1. have you tried training with QLoRa, and not just LoRa ? are you considering adding it to the repo ?

2. the example usage (https://github.com/shibing624/CodeAssist#train-wizardcoder-model) is without ddp, only dp.
   are you sure this is the optimal setting? We got significantly higher training rates with ddp

[shibing624]

1. qlora will add, next week. qlora done in my repo: https://github.com/shibing624/MedicalGPT
2. use torchrun

[mrT23]

The base example is bad (no ddp)

cd examples
CUDA_VISIBLE_DEVICES=0,1 python training_wizardcoder_mydata.py --do_train --do_predict --num_epochs 1 --output_dir outputs-wizard --model_name WizardLM/WizardCoder-15B-V1.0

and there are some discrepancies in the code, that need adjustments to ddp, for example doing 'do_predict' just on the main process.

anyway, thanks for the answer, and the great code.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.(由于长期不活动，机器人自动关闭此问题，如果需要欢迎提问)