Exception: Current loss scale already at minimum - cannot decrease scale anymore

[Z-eloto]

Thank you for sharing your codes.
When running gpt2/kd/kd_medium.sh on 2*3090, the program encountered this error. What should I do, such as adjusting the learning rate?

[shiboyu1999]

You can use fp32 to train the model or decrease the batch size to 1.

[Z-eloto]

You can use fp32 to train the model or decrease the batch size to 1.

Thanks. I will try it. :)

[t1101675]

You can also try using bfloat16 by replacing ds_config_zero1_fp16.json in this line with ds_config_zero1_bf16.json.