Update model-client docs to include external LLM support via OpenAI c…

[vballoli]

…lient

Why are these changes needed?

Adds documentation for users that need to use different LLMs through the OpenAI API

Related issue number

"Addresses: #4117"

Checks

• I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
• I've added tests (if relevant) corresponding to the changes introduced in this PR.
• I've made sure all auto checks have passed.

[ekzhu]

Thanks for the speedy turn around

[ekzhu]

1. Can you run the gemini example to generate some result so we can know for sure it works? Because I cannot test it.

2. Can you create a separate section under Azure Open AI, call it "OpenAI-Compatible Models", and move the Gemini example there.

[vballoli]

Yes, I did run the simple example and it worked. I've also run the following code to make sure it worked and it did.

from autogen_agentchat.agents import CodingAssistantAgent
from autogen_agentchat.task import TextMentionTermination
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_core.components.models import OpenAIChatCompletionClient

planner_agent = CodingAssistantAgent(
    "planner_agent",
    model_client=OpenAIChatCompletionClient(model="gemini-1.5-flash", base_url="https://generativelanguage.googleapis.com/v1beta/", api_key="API_KEY", model_capabilities={
        "vision": True,
        "function_calling": True,
        "json_output": True,
    },),
    description="A helpful assistant that can plan trips.",
    system_message="You are a helpful assistant that can suggest a travel plan for a user based on their request.",
)

local_agent = CodingAssistantAgent(
    "local_agent",
    model_client=OpenAIChatCompletionClient(model="gemini-1.5-flash", base_url="https://generativelanguage.googleapis.com/v1beta/", api_key="API_KEY", model_capabilities={
        "vision": True,
        "function_calling": True,
        "json_output": True,
    },),
    description="A local assistant that can suggest local activities or places to visit.",
    system_message="You are a helpful assistant that can suggest authentic and interesting local activities or places to visit for a user and can utilize any context information provided.",
)

language_agent = CodingAssistantAgent(
    "language_agent",
    model_client=OpenAIChatCompletionClient(model="gemini-1.5-flash", base_url="https://generativelanguage.googleapis.com/v1beta/", api_key="API_KEY", model_capabilities={
        "vision": True,
        "function_calling": True,
        "json_output": True,
    },),
    description="A helpful assistant that can provide language tips for a given destination.",
    system_message="You are a helpful assistant that can review travel plans, providing feedback on important/critical tips about how best to address language or communication challenges for the given destination. If the plan already includes language tips, you can mention that the plan is satisfactory, with rationale.",
)

travel_summary_agent = CodingAssistantAgent(
    "travel_summary_agent",
    model_client=OpenAIChatCompletionClient(model="gemini-1.5-flash", base_url="https://generativelanguage.googleapis.com/v1beta/", api_key="API_KEY", model_capabilities={
        "vision": True,
        "function_calling": True,
        "json_output": True,
    },),
    description="A helpful assistant that can summarize the travel plan.",
    system_message="You are a helpful assistant that can take in all of the suggestions and advice from the other agents and provide a detailed tfinal travel plan. You must ensure th b at the final plan is integrated and complete. YOUR FINAL RESPONSE MUST BE THE COMPLETE PLAN. When the plan is complete and all perspectives are integrated, you can respond with TERMINATE.",
)

termination = TextMentionTermination("TERMINATE")
group_chat = RoundRobinGroupChat(
    [planner_agent, local_agent, language_agent, travel_summary_agent], termination_condition=termination
)

import asyncio
result = asyncio.run(group_chat.run(task="Plan a 3 day trip to Nepal."))
print(result)

TaskResult(messages=[TextMessage(source='user', models_usage=None, content='Plan a 3 day trip to Nepal.'), TextMessage(source='planner_agent', models_usage=RequestUsage(prompt_tokens=0, completion_tokens=0), content="## 3-Day Nepal Adventure: Kathmandu & Beyond\n\nThis itinerary focuses on Kathmandu's cultural richness and includes a day trip to the scenic Pokhara valley. It's a good starting point, and you can tailor it based on your interests:\n\n**Day 1: Kathmandu - Cultural Immersion**\n\n* **Morning:** Arrive at Tribhuvan International Airport... 

[vballoli]

Updated the docs

[ekzhu]

@vballoli thanks for the fast turn-around and edits, though after discussion with our team we decided to make the list of supported models more precise. I will submit a separate PR for this issue.

[ekzhu]

Will address in a separate PR. Thank you @vballoli for the work! It helps to inform us the newly added OpenAI endpoint support from Gemini.

[vballoli]

Happy to help! Looking forward to contributing more to autogen

[ekzhu]

@vballoli I opened a separate issue: #4174

Meanwhile, if you are interested in maintaining a Gemini client as a community autogen package, you can take a look at the instruction here: https://microsoft.github.io/autogen/dev/user-guide/extensions-user-guide/index.html#list-of-community-projects

[vballoli]

Yep this looks good - I'll follow up on creating this over the weekend