llama-cpp-python server not working with "OpenAI API Connections" #2477
nick-tonjum
started this conversation in
General
Replies: 1 comment
-
Hi I am having the same issue regarding the integration of llama-cpp-python into o-wu. It seems to me that the definition of the api standard is different for o-wu than it is for llama-cpp-python, i.e. llama.cpp. Here is what I was expecting when it said OpenAI compatible api. https://llama-cpp-python.readthedocs.io/en/latest/server/#configuration-and-multi-model-support. However, when looking at how they discover the models in o-wu, it is more in line with https://platform.openai.com/docs/api-reference/models/list where the models should be in the |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello!
I'm having an issue with the 'OpenAI API Connections' in the settings.
I currently have my OpenAI api key working there perfectly, but when I try to add the address for a llama-cpp-python server it isn't listing the models whatsoever, despite saying it successfully added.
My open-webui instance is running in a docker container, so this is what I have entered:
http://host.docker.internal:4883/v1
no API key since it's internal, but still doesn't work if I have one set. I also tried localhost, 0.0.0.0, 127.0.0.1, and even tried opening it up to my public IP and it didn't work. I can manually GET the models at http://host.docker.internal:4883/v1/models and it works without issue.
I also know it's not an issue with host.docker.internal, as my ollama server itself is running on it and I have my ollama server pointed to host.docker.interal:11434.
I'm stuck.
Thanks!
Beta Was this translation helpful? Give feedback.
All reactions