-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VRAM memory leak for Refact.AI 1.6B #332
Comments
Thanks for reporting. I don't think we do anything that can cause memory leaks. Hmm maybe it's the torch version or cuda version or something like this 🤔 |
Same with deepseek-coder/1.3b/base (finetune), start at ~3GB and after one hour, up to 7gb. Os: linux mint |
I'll try to reproduce |
I left 1.6b (regular backend) for a day, memory settled on 6.19 Gb of memory RAM. I additionally sent 750 completion requests today and it's still 6.19 GB. I don't say there's no leak, I can only say I tried and I don't see a leak in my setup 🤔 Not sure what to do... |
Called for help from @mitya52 |
@d3v2a it looks like normal behavior. On start model allocates 3gb but when you start using it with large context (on large files for example) it allocates additional memory for it. I see no memory leaks with your case. |
hmm now I see 11.9Gb on my setup 🤔 |
The problem seems not to be present in the last version |
Cool! |
I've updated to latest sha256:f1968874 and it works ok. Usage stabilized around 9.6 on 10GB VRAM GPU and there are no issues as it seems. |
Windows 11 fully updated.
WSL2 updated.
Docker Desktop for Windows latest , GPU works in docker. nvidia-sli reports the GPU fine.
Nvidia Cuda 12.2 Toolkit
Newest Nvidia drivers.
RTX 3080 10GB VRAM.
AMD R5800X3D 32GB RAM
No other GPU software running.
At first all looks good, model loads and is serving, but after some time memory utilization grows to 10GB and then GPU load stays at 100% for prolonged times, model times out I can only restart the docker container to fix it.
Actually it rises to 10GB of VRAM use pretty quickly. This is for 1.6B Refact.ai model.
Docker runs 'thenlper/gte-base' as well. When I delete it to gain a little VRAM, the responsiveness comes back for just a couple of queries more.
JetBrains IDEA Refact.AI plugin.
The text was updated successfully, but these errors were encountered: