LiteLLM Integration in the Backend

[justinh-rahb]

Hi everyone,

I'd like to propose integrating LiteLLM directly into the Ollama WebUI project. This would allow us to have greater flexibility in the OpenAI-compatible external APIs we can use, as well as giving us more control over how we use and configure those APIs.

There are a few different ways we could go about integrating LiteLLM:

1. Directly into the backend as a Python module: This would involve adding LiteLLM as a dependency in the backend code, and then importing and using it as needed. This would be the most straightforward option, but it may require some refactoring of the current backend code to properly incorporate LiteLLM.
2. Using the pre-built proxy container: LiteLLM provides a pre-built Docker container that acts as a reverse proxy to any compatible API. We could use this container in conjunction with a frontend settings dialog to allow users to configure the LiteLLM config via its API. This would give us more flexibility than the first option.

Regardless of which approach we take, I believe that integrating LiteLLM into Ollama WebUI would provide significant benefits in terms of flexibility and control. I'd love to hear everyone's thoughts on this proposal!

[tjbck]

I've actually been thinking about incorporating LiteLLM directly to the webui as well, this will be helpful for a lot of people!

[justinh-rahb]

Thinking about it more, you could probably actually remove a significant amount of code from the project as is by making LiteLLM the core piece that handles all chat interactions, including Ollama.

[justinh-rahb]

Ollama now has OpenAI API compatibility: https://github.com/ollama/ollama/releases/tag/v0.1.24

I've grown even more convinced that the project could significantly simplify its codebase by exclusively using one API middleware library. This would allow for the efficient aggregation of multiple endpoints within the interface. Observing that a vast majority of inference backends are aligning with the OpenAI API specification, it seems feasible to make the UI backend-agnostic by solely supporting OAI. This move could greatly reduce complexity and increase the system's flexibility.

"The quality of a project is not measured by the features it adds, but by the unnecessary code it eliminates."

-Somebody

[justinh-rahb]

Also @tjbck I kinda dig "One UI" now, it's fitting if you adopted this approach.

[NinjaPerson24119]

This sounds like the easiest thing to do IMO. Then you're not tightly coupled to a huge dependency either.

[arthurGrigo]

+1

Is there any progress going in this direction?