-
Notifications
You must be signed in to change notification settings - Fork 38
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
VLLM assert isinstance(self.device, list) error #27
Comments
and... AttributeError: 'UserDict' object has no attribute 'instance' |
This package does not seem to implement parallel reasoning at all? According to my observation, it actually just repeats the same prompts to each model instead of sharding the prompts to the model, so there is no speed bonus for parallel reasoning at all. |
from datadreamer.llms import ParallelLLM,VLLM,LLM,HFTransformers
VLLM(Mistral-7B-Instruct-v0.2", device=[1]).model
will rasie device error
File ~/autodl-tmp/conda/envs/wslconda/lib/python3.10/site-packages/datadreamer/llms/vllm.py:74, in VLLM.model(self)
72 env = os.environ.copy()
73 print(self.device)
---> 74 assert isinstance(self.device, list)
75 env.update(get_device_env_variables(self.device))
76 kwargs = self.kwargs.copy()
The text was updated successfully, but these errors were encountered: