-
Notifications
You must be signed in to change notification settings - Fork 398
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
gptq 4bit avg loss is large #643
Comments
Make sure you have proper dataset that is similar to the training dataset. |
@moseshu Have you figure this problem out? |
Based on what you posted, can't tell if quant is good or not. Do ppl post quant and compare running avg loss of early vs later layers. |
I user aotugptq convert blfloat model to in4 the avg loss is a bit larger than int 8.
Model is Mixtral-8X7B
int8 loss is almost 0.0004
it means that the loss of int4 is more serious?
The text was updated successfully, but these errors were encountered: