Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Switch to vLLM #58

Open
svenseeberg opened this issue Oct 9, 2024 · 1 comment
Open

Switch to vLLM #58

svenseeberg opened this issue Oct 9, 2024 · 1 comment
Assignees
@svenseeberg
Labels
component:chat Chat Back End enhancement New feature or request
Milestone
v3.2 Improvements...

Comments

@svenseeberg
Copy link
Member

Ollama has limitations, for example with available models and loading them. We may want to switch to vLLM.
@svenseeberg svenseeberg added this to the v3 Basic Answer Retrieval milestone Oct 9, 2024
@svenseeberg svenseeberg self-assigned this Oct 9, 2024
@svenseeberg svenseeberg added component:chat Chat Back End enhancement New feature or request labels Oct 9, 2024
@svenseeberg svenseeberg mentioned this issue Oct 9, 2024
Open
@svenseeberg svenseeberg changed the title Swtich to vLLM Switch to vLLM Oct 16, 2024
@svenseeberg
Copy link
Member Author

Almost done: https://git.verdigado.com/verdigado-Privileged/Salt/pulls/2226

However, we need to find models that fit into the graphic card memory.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@svenseeberg svenseeberg

Labels
component:chat Chat Back End enhancement New feature or request
Projects
None yet
Milestone
v3.2 Improvements of Basic Features
Development

No branches or pull requests
1 participant
@svenseeberg