Dynamically generate agent systems in AutoGen #4226
Conversation
yeandy
changed the title
There was a problem hiding this comment.
Copilot reviewed 9 out of 9 changed files in this pull request and generated no suggestions.
| TLDR: | ||
| This is a feature to use a meta-agent to generate new agent systems. For example, this agent system was discovered and written entirely by an agent using o1-preview: | ||
| ``` | ||
| def forward(self, task, model_client_kwargs): |
There was a problem hiding this comment.
would love to get rid of this as I feel like its still rooted in the ADAS implementation.
| return documentation | ||
|
|
||
|
|
||
| def get_autogen_documentation() -> List[str]: |
There was a problem hiding this comment.
i love you dynamically pulling this. Do we need to think about matching versions of what it pulls to the version of code you have?
There was a problem hiding this comment.
We are pulling documentation under https://github.com/microsoft/autogen/tree/main/python/packages/autogen-core/docs/src/user-guide/core-user-guide, and using the default main branch, which has the most up-to-date API. I assume anyone who changes the API behavior will also change the documentation, as well as all example code (including this one) that would be affected.
| Note: If you add a new agent system after you’ve started generating new Agent Systems (next [section](#generate-new-agent-systems)), the meta-agent will not pick up this new seed agent system. This is because it will try to first detect the results file defined by the expr_name flag, and reference that file for the agent archive, instead of the `adas_prompt.py` file. | ||
|
|
||
| ### Generate new Agent Systems | ||
| #### Prepare your dataset |
There was a problem hiding this comment.
what about including an example already that works out of the box?
There was a problem hiding this comment.
The current instructions say to clone the https://github.com/ShengranHu/ADAS repo and use the https://github.com/ShengranHu/ADAS/blob/main/dataset/drop_v0_dev.jsonl.gz dataset that is present in that repo. Did you mean that we should add a dataset file directly to the autogen repo? i.e. copy drop_v0_dev.jsonl.gz to the autogen repo, or maybe create a different toy dataset?
| # Install autogen-core and autogen-ext in editable mode | ||
| pip install -e packages/autogen-core | ||
| pip install -e packages/autogen-ext | ||
| pip install -r packages/autogen-core/samples/adas/requirements.txt |
There was a problem hiding this comment.
id also include pandas and numpy as you are using them in the example code
There was a problem hiding this comment.
I have numpy in the requirements.txt. Did you find yourself to have to separately install pandas?
|
|
||
| This should be passed as a JSON string to the `meta_agent_model_config` flag. | ||
| ```bash | ||
| --meta_agent_model_config='{"api_version": "2024-08-01-preview", "azure_endpoint": "https://<user>-aoai1.openai.azure.com/openai/deployments/o1-preview/chat/completions?api-version=2024-08-01-preview", "model_capabilities": {"function_calling": false, "json_output": false, "vision": false}, "azure_ad_token_provider": "DEFAULT", "model": "o1-preview-2024-09-12"}' |
| - `base_agent_model_config`: JSON string of the `AzureOpenAIChatCompletionClient` settings for the Base Agent. | ||
| - `n_generation`: number of generations of new agents that the meta-agent tries to discover | ||
| - `expr_name`: name of the output file containing both the original/seed and newly generated agent systems, as well as their fitness scores. | ||
| - `max_workers`: the number of threads to spin up in a `ThreadPoolExecutor`, to parallelize the execution of the particular Agent System that is currently being evaluated. |
There was a problem hiding this comment.
whats the default value and what is the recommended value as per CPUs? Can we dynamically set that based on CPUs? Is it only influenced by CPUs or also model capacity?
There was a problem hiding this comment.
Default is 48, but that actually gets adjusted based on the number of examples and the multiprocessing flag as well https://github.com/yeandy/autogen/blob/yeandy_adas/python/packages/autogen-core/samples/adas/adas.py#L138.
It currently does do any smart detection based on # of CPUs. If you know that number (i.e. 32 cores), then of course you can set the value of max_workers. Though the "optimal" value also depends on the tokens/sec limit on your particular model deployment, where you may still need to cap the number of max_workers to say 10 if the number of parallel LLM calls made be the Agent System being evaluated is high.
However, in practice, I've found for some Agent Systems that running with max_workers > 1 causes some hanging issues with the AutoGen messaging. i.e. the E2E process doesn't terminate. For these, I had to manually set max_workers=1. See the "Troubleshooting" section in the README for the details. Unfortunately, the reason for this is unknown. Will need insight from other devs on what is going on.
| ```bash | ||
| --base_agent_model_config='{"api_version": "2023-03-15-preview", "azure_endpoint": "https://<user>-aoai1.openai.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2023-03-15-preview", "model_capabilities": {"function_calling": true, "json_output": true, "vision": true}, "azure_ad_token_provider": "DEFAULT", "model": "gpt-4o-2024-08-06"}' | ||
| ``` | ||
| ### Run ADAS |
There was a problem hiding this comment.
Running adas.py does seem to generate standard output and standard error output. Strangely it seems like the output in the error stream is of type info. I recommend to consolidate and also give users the option to adjust the logging threshold. I feel like we want to aim at INFO including all relevant output that allows a user to understand what ADAS is doing. Put everything else into DEBUG. Id recommend to ensure lengthy prompts are in DEBUG, where as the that the LLM was queried and gist of the prompt should be in INFO.
Moved. Thanks! |
Why are these changes needed?
Version 1 of the ADAS implementation in AutoGen. Please read the
README.mdin this PR for details.Related issue number
This is the PR for the feature initially discussed here #4569 and #4486
Checks