Pre-call hooks?

[twardoch]

In a sense, LiteLLM (the proxy) is a way to create an OpenAI-compatible endpoint. I'd like to deploy my own OpenAI-compatible endpoint on some serverless provider, but in such a way that whenever I prompt my own endpoint, "something happens" and only then I'm routing you the LLM.

For example, I might want to do some prompt pruning or do my own code-based choice of the right model to call.

Or maybe I want to call my own RAG. So basically I might want an OAI-compatible endpoint that calls something that's beyond just the LLM. A simple way to do it might be be exposing some pre-call hooks. My endpoint accepts a prompt but then it actually calls my own RAG and only with the enriched prompt it calls the LLM.

Is this already possible?

[twardoch]

Ok, found it at https://litellm.vercel.app/docs/observability/custom_callback#callback-functions

[ishaan-jaff]

Hi @twardoch can we hop on a call sometime this week to learn how we can improve litellm for you ? What's the best email to send an invite to ?

If it's easier here's my cal: https://calendly.com/d/4mp-gd3-k5k/litellm-1-1-onboarding-chat

[cduk]

I'm trying to do the same thing and saw this: https://litellm.vercel.app/docs/proxy/logging#custom-callback-class-async

What was not clear to me from the documentaiton is where the custom callback class file should be located (I'm trying to keep logic on the proxy server).

EDIT: from this other example: https://docs.litellm.ai/docs/proxy/guardrails/custom_guardrail

I see they suggest putting it in /apps dir in the docker image. So I will try that.