How can we support LLM Route to Generalize the LLM Provider

[Krishanx92]

Currently support LLMProviders, such as OpenAI, Mistral, Azure, and others has its own specific configurations and implementations. For instance, in the case of OpenAI, the model details are defined in the request body. Additionally, rate-limiting parameters, such as prompt tokens, completion tokens, and total tokens, are also in body.

Current approach involves handling these provider-specific configurations and logic (e.g., transformations) inside translators and external processors (extproc). While this approach works, it tightly couples the implementation to predefined providers and may limit flexibility for supporting custom LLMs.

To make the implementation more flexible and extensible, Can we explore generalizing the approach by storing provider-specific and rate-limiting information directly within the LLMRoute resource? Then will it become LLM Provider rather than LLM Route?

Key Considerations could be
Generalization: LLMRoute that supports diverse providers(Different Models) and rate-limiting use cases while remaining extensible?
Rate-Limiting: Should LLMRoute include fields for prompt tokens, completion tokens, and total tokens, or should these remain provider-specific?

Looking forward to thoughts and suggestions.

[mathetake]

let's discuss post v0.1.0 release!