Budget Management - Limit token sizes

[kmesiab]

Some diffs can be quite large. Sometimes those large diffs include changes that are not relevant to a code review, like dependency updates.

Sending these diffs to OpenAI will consume credits unecessarily.

Implement a relief valve and budget manager for token usage.
The ability to set a token limit.

This article from Microsoft introduces a strategy for prompt compression with no loss.

https://arxiv.org/abs/2310.05736

[ishaan-jaff]

Hi @kmesiab I'm the maintainer of LiteLLM https://github.com/BerriAI/litellm we allow you to do cost tracking for 100+ LLMs

Usage

Docs: https://docs.litellm.ai/docs/#calculate-costs-usage-latency

from litellm import completion, completion_cost
import os
os.environ["OPENAI_API_KEY"] = "your-api-key"

response = completion(
  model="gpt-3.5-turbo", 
  messages=[{ "content": "Hello, how are you?","role": "user"}]
)

cost = completion_cost(completion_response=response)
print("Cost for completion call with gpt-3.5-turbo: ", f"${float(cost):.10f}")

We also allow you to create a self hosted OpenAI Compatible proxy server to make your LLM calls (100+ LLMs), track costs, token usage
Docs: https://docs.litellm.ai/docs/simple_proxy

I hope this is helpful, if not I'd love your feedback on what we can improve

[kmesiab]

This looks great @ishaan-jaff , I will give it a go!