how text_completion(model) knows if the passed model is a text-completion or chat-completion mode?

[solyarisoftware]

Here to ask clarifications for a possible bug.

In my application I'm successfully using LiteLLM text_completion() method:

https://github.com/solyarisoftware/prompter.vim/blob/master/python/llm.py#L35

BTW, in my specific use cases / environment, I'm using an Azure deployment called "azure/gpt35-turbo" that, as the name suggests, is enabled by the OpenAI chat-completion mode model GPT3.5-Turbo.

text_completion() set the prompt argument as a system prompt. Fine.

My question is:
What if the model passed in enabled by a TEXT-MODE model?
By example, just to refers to OpenAI LLM provider TEXT-MODE model (compatible with legacy Completions endpoint and not Chat Completions.), examples are:

• text_da_vinci_003
• gpt-3.5-turbo-instruct
• etc.

Reading the text_completion() implementation code:

litellm/litellm/main.py

Line 1861 in f6d40c2

def text_completion(

I do not understand how the function could know is the model is a chat-completionor a text-completion model and consequently call the correct legacy Completion endpoint and not Chat Completion.

My expectation/wish is that text_completion() would work also for ANY model (also for all NO-OpenAI text-completion models). I fair text_completion() lacking of an argument that specify if the model is chat-completion or text-completion, fails for text-completion models. Isn't it?.

Please let me know
giorgio

Twitter / LinkedIn details

twitter: @solyarisoftare linkedin: www.linkedin.com/in/giorgiorobino

[krrishdholakia]

moving this to a discussion.

Is there a specific model you have in mind as you say this? @solyarisoftware

[solyarisoftware]

Hi @krrishdholakia

• TEXT-COMPLETION-MODE MODELS

  examples of text-mode models form OpenAI provider are:

  • text-da-vinci_003
  • gpt-3.5-turbo-instruct

  BTW, For the full list of OpenAI text-completion models (endpoint: https://api.openai.com/v1/completions), see details here:
  https://platform.openai.com/docs/guides/text-generation/text-generation-models

  Complete list:

  • Updated legacy models (2023) gpt-3.5-turbo-instruct, babbage-002, davinci-002
  • Legacy models (2020–2022) text-davinci-003, text-davinci-002, davinci, curie, babbage, ada

  But I think there are many other provider models that supply text-mode (no-chat-mode, requiring OpenAI-like legacy
  Completions endpoint).

• CHAT-COMPLETION-MODE MODELS

  • OpenAI Newer models (2023–) | gpt-4, gpt-4 turbo, gpt-3.5-turbo (through endpoint: https://api.openai.com/v1/chat/completions)

[solyarisoftware]

So, take for example this basic program:

# !pip install -U litellm
# https://docs.litellm.ai/docs/tutorials/azure_openai
# https://docs.litellm.ai/docs/tutorials/text_completion

from litellm import text_completion

prompt = 'Write a tagline for a traditional italian tavern, as a sentence is in Italian language, without quotes'

response = text_completion(
    model="azure/gpt-35-turbo-0613",
    prompt=prompt,
    temperature=0.7,
    max_tokens=100,
    stop=None
)

print(prompt)
print('=>')
print(response['choices'][0]['text'])

In the above case, the model is an Azure deployment associated with the Openai model type GPT3.5-Turbo, version "0613", as the name suggests. Now this model is what I call a CHAT completion type model, and to get a TEXT completion, the text_completion() litellm function rightly put the prompt as a SYSTEM PROMPT message (role: system). That's working fine.

But what if I'm passing to the text_completion() a deployment or model that is naively a TEXT completion type model as a gpt-3.5-turbo-instruct ?
Reading the text_completion() code I do not understand how the function could determine if the model is a TEXT completion or chat-completion!

I hope to have been more clear in the above explanation.
Looking forward your feedback
Thanks you
giorgio