Some questions about Auto Mixed Precision Training /expected scalar type Float but found Half

[xxw11]

1.当我使用如下配置时会发生错误
fp16 = dict(
mode=AMP_TYPE.NAIVE
)

但是我如果切换fp16的类型为AMP_TYPE.TORCH将会正常运行，
fp16 = dict( mode=AMP_TYPE.TORCH )
我不知道这可能是哪方面的原因？

2.似乎fp16和zero不能一起使用，
我会报如下错误
It is not allowed to set fp16 and zero configuration in your config file at the same time
但是当我注释掉关于fp16的设置会出现如下错误

下面是我的部分设置
`
BATCH_SIZE = 8
SEQ_LEN = 2048
NUM_EPOCHS = 50
TENSOR_PARALLEL = 4

zero = dict(
level=2,
dynamic_loss_scale=True,
overlap_comm=True,
clip_grad=1.0,
cpu_offload=False,

)

gradient_accumulation = 5

optimizer = dict(
type=SGD,
lr=0.00015,
weight_decay=1e-2,
)

loss = dict(
type=GPTLMLoss,
)

model = dict(
type=gpt2_Y,
checkpoint=True,
)

parallel = dict(
pipeline=1,
tensor=dict(size=TENSOR_PARALLEL, mode='2d'),
)
`

[ver217]

Hi, we are trying to reproduce your problem. Could you tell the exact version of your colossalai?

[xxw11]

colossal version 0.0.1

[ver217]

Hi, we have fixed this bug since the 0.0.2 version. Could you give it a try? You can install from PYPI or install from source using the latest version.

[xxw11]

Thank you so much. Now I can use AMP_TYPE.NAIVE successfully.

[xxw11]

Dear developer:
But I still can't use ZeRO

[xxw11]

Dear developer:
But I still can't use ZeRO

[xxw11]

FrankLeeeee, you are so nice, friendly and helpful.
After I tried, I found you are right. But I don't understand. Is this a feature?

[FrankLeeeee]

ZeRO currently does not work with gradient accumulation well. We are trying to reconstruct zero for better speed and compatibility. We will release this in the next version.

[FrankLeeeee]

I have created an issue in #247 , and we will update our progress accordingly. You can keep an eye on this issue if you want to :)

[xxw11]

ColossalAI is the best HPC&AI framework I've ever used. It is easy but powerful. It's a really great job. I can't wait to see your next version.

[FrankLeeeee]

Thanks for your compliment! Really happy to hear that and we will keep working on this 🚀