-
Notifications
You must be signed in to change notification settings - Fork 136
-
Notifications
You must be signed in to change notification settings - Fork 136
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
quantization and gpu acceleration of the quantized model. #88
Comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Wasn't sure if this was technically a new issue but just in case I'm reposting here:
I figured out how to dynamically quantize the instructor-xl model, but at the point that it's supposed to create the embeddings, i want it to use gpu acceleration (cuda) just like it does when I use the float32 version of the model. Is that possible? If I understand the comments above, it's not? What about quantizing the model beforehand NOT using the "dynamic" method? I've been struggling with this for months so any help would be much appreciated. The link above is to a discussion back in 2021 and "seek other solutions" doesn't point me in the right direction so...I'm looking at bitsandbytes but couldn't find a solution either... Here is the portion of the script I'm trying to use:
Also, when I try to use instructor-xl with the huggingfaceembeddings class specifically (i.e. not using
embeddings = qmodel.encode([[instruction,sentence]])
it won't work either...The text was updated successfully, but these errors were encountered: