Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

deciduus' development pulls ~ IDE terminal compatibility, docker quality of life, prompt optimizations, execution handling & style encoding, NVIDIA CUDA (GPU support) ~ #19

Open
wants to merge 12 commits into
base: main
Choose a base branch
from

Conversation

deciduus
Copy link

@deciduus deciduus commented Jul 30, 2024

I'm new here, but continuing to play around and tweak but I'm submitting a pull request as these seem like good QoL for everyone.

deciduus added 3 commits July 29, 2024 21:51
added pycharm ide running compatibility
slight edit of prompt
slight revision of prompt
Copy link

@sanduabey sanduabey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changes look good.

deciduus added 2 commits July 30, 2024 17:54
refined prompts for clarity and conciseness adding some new steps for logic and reducing chance of subordinate looping
increased main prompt timeout to 2 min so it won't run so quick for so little value
…jects.

made edits to the docker.py to include a health check (threaded), it does not impact framework flow. also added a check and reconnect mechanic for the docker due to some issues where it would disconnect, try terminal commands, and get no response (wasting api responses). the health check thread also follows up to close the docker container upon force close in IDE, which helps save computer resources.
knowledge tool was updated to include a top 3~ hits for wikipedia content to build the contextual awareness.
timed input was adjusted upward for testing.
@deciduus
Copy link
Author

I think the latest commit from a few minutes ago has several useful features that could benefit the main project.

Docker Enhancements:

  • Added a threaded health check in docker.py to monitor container status without impacting framework flow.
  • Implemented a check and reconnect mechanism to handle Docker disconnections, preventing wasted API responses.
  • The health check thread ensures Docker containers are properly closed upon force close in IDE, conserving computer resources.

Knowledge Tool Update:

  • Enhanced the tool to include the top ~3 hits from Wikipedia, improving contextual awareness.

Timed Input Adjustment:

  • Increased the timeout duration for testing purposes.

@deciduus deciduus changed the title edited main so that the framework can be ran in IDE (like pycharm), slight prompt revisions (and fix typos) to decrease change of looping subordinates. deciduus' development & improvement pulls (IDE terminal compatibility, docker quality of life, prompt optimizations) Jul 31, 2024
…d errors (lack of gpu resources), to prevent the error in tool.

docker wait time was decreased as the connection doesnt take that long
print style added encoding utf-8 to prevent read errors, this resulted in many terminal tasks that were previously "erroring" due to simply taking too long or having a bad symbol to actually function better.
@deciduus
Copy link
Author

deciduus commented Aug 1, 2024

  • Code Execution Tool: Modified to handle the absence of GPU resources more gracefully, preventing related errors in tool usage.
  • Docker Setup: Reduced wait time during container startup, as the connection establishes faster than initially expected.
  • Print Style: Updated to use UTF-8 encoding, resolving issues with terminal outputs that previously errored due to extended execution times or unexpected symbols. This change improved the reliability of terminal tasks.

@deciduus deciduus changed the title deciduus' development & improvement pulls (IDE terminal compatibility, docker quality of life, prompt optimizations) deciduus' development & improvement pulls (IDE terminal compatibility, docker quality of life, prompt optimizations, execution handling & style encoding) Aug 1, 2024
edited docker file and readme to allow for nvidia cuda support, giving access to gpu resources.

added readme steps to enable the GPU compatibility setup
@deciduus
Copy link
Author

deciduus commented Aug 1, 2024

undid the code execution tool edit from before

edited docker setup to change to nvidia cuda, allowing gpu resource access for docker

added readme steps at bottom elaborating on the cuda setup (ubuntu and docker)

  • debian is commented out and easily swapped back to - im planning to do a auto-fallback to debian if cuda doesnt work or isn't compatible for whatever reason

…d perms for container.

added commented out code in dockerfile for trying to import torch and numpy at start, but currently having issues with it initiating (but pushing so others have it incase they want to try to fix)
@deciduus
Copy link
Author

deciduus commented Aug 1, 2024

Updated README with comprehensive instructions for setting up NVIDIA CUDA, Ubuntu, permissions, and Docker configurations.

Note: Dockerfile includes commented-out code for the initial install of Numpy and PyTorch, but I haven't had it work yet.

@deciduus deciduus changed the title deciduus' development & improvement pulls (IDE terminal compatibility, docker quality of life, prompt optimizations, execution handling & style encoding) deciduus' development & improvement pulls ~ IDE terminal compatibility, docker quality of life, prompt optimizations, execution handling & style encoding, NVIDIA CUDA (GPU support) ~ Aug 1, 2024
@deciduus deciduus changed the title deciduus' development & improvement pulls ~ IDE terminal compatibility, docker quality of life, prompt optimizations, execution handling & style encoding, NVIDIA CUDA (GPU support) ~ deciduus' development pulls ~ IDE terminal compatibility, docker quality of life, prompt optimizations, execution handling & style encoding, NVIDIA CUDA (GPU support) ~ Aug 1, 2024
@drhouse
Copy link

drhouse commented Aug 2, 2024

edited docker setup to change to nvidia cuda, allowing gpu resource access for docker

looks interesting, I'm curious does this mean : 

i) agent-zero can run in Ubuntu with a GUI ii) does agent-zero itself run using cuda or iii) is it so one can work on nvidia cuda related code projects?

I started installing Linux Mint to see if I can have Agent-Zero use a GUI environment, like controlling Chrome with Puppeteer... instead of Ubuntu can your fork also use Linux Mint?

@deciduus
Copy link
Author

deciduus commented Aug 2, 2024

edited docker setup to change to nvidia cuda, allowing gpu resource access for docker

looks interesting, I'm curious does this mean : 

i) agent-zero can run in Ubuntu with a GUI ii) does agent-zero itself run using cuda or iii) is it so one can work on nvidia cuda related code projects?

I started installing Linux Mint to see if I can have Agent-Zero use a GUI environment, like controlling Chrome with Puppeteer... instead of Ubuntu can your fork also use Linux Mint?

my unprompted reply:
yes the purpose was giving the container access to the gpu through ubuntu. This version worked for me with GPU usage using pytorch packages. That said sometimes it still throws a GPU driver not found seemingly related to composing the container, which seems like a whole other can of worms. I'm not an expert here but prior to digging into compose and such i did see confirmed gpu usage and it generated cat pics and i didnt compose or run -gpus all commands (but because of error i have been attempting to see if a more persistent configuration is plausible). * I'm learning a lot of this as I go so I have limited exposure to these other tools at this stage just sharing my perspective a bit for context.

gpt 4o helping me objectify the description above:

Yes, the setup aims to enable GPU access within the container using NVIDIA's CUDA, allowing for the utilization of GPU resources. This configuration enables Agent-Zero to leverage GPU acceleration for tasks such as running PyTorch models, significantly enhancing performance for machine learning workloads.

Key Points:
Agent-Zero with GUI on Ubuntu:
While Agent-Zero can technically run in an Ubuntu environment with a GUI, this specific setup focuses on enabling GPU access rather than GUI capabilities. However, if your goal is to interact with GUI applications like Chrome via Puppeteer, additional configuration would be necessary.

Usage of CUDA:
The setup allows for CUDA usage within the container, providing GPU acceleration for applications like PyTorch. Agent-Zero itself doesn't inherently require CUDA, but this setup supports GPU-accelerated operations.

Linux Mint Compatibility:
Linux Mint, being based on Ubuntu, should generally be compatible with this setup. Similar configuration steps can be followed for GPU access and CUDA support.

Additional Note:
I'm currently working on persistent configuration via Docker Compose (docker-compose.yml). This involves fine-tuning the setup to consistently access the GPU and minimize issues like "GPU driver not found" errors. While the GPU setup is functional and has been successfully used for generating outputs like images, refining the Docker Compose setup aims to make this configuration more stable and reliable.

- build.txt same ^
- code execution tool got elaborate updates for prompt-handling pass-back, idle timeout, comparing old and new outputs to prevent spamming user with identitical terminal commands, and a summarized output feature to prevent slamming API with too many tokens on big terminal tasks, also the terminal will retain control and the agent will not check in unless idle proc (about 30 seconds)
-docker.py sleeps slightly longer anticipating some may take longer to spin up the new container.
-docker compose has been added for the user to create the nvidia image - please see readme for detailed instructions
-dockerfile has been adjusted, new container (old is commended out), timezone check for automatic selection on image compose. organized a bit.
-main.py adjusted the commented out docker names just for clarity
-adjusted rate_limiter.py to fix a bug when the agent was doing long tasks and lots of calls
-added intensive README section for ubuntu and nvidia cuda runtime setup and troubleshooting.
-timeout_input was reverted to original value
@deciduus
Copy link
Author

deciduus commented Aug 4, 2024

This update introduces significant enhancements, focusing on enabling NVIDIA CUDA support for GPU access, which now allows for running GPU-intensive tasks such as deep learning and other computational workloads. This includes setting up the necessary infrastructure for efficient GPU utilization. Alongside this, comprehensive improvements have been made to streamline the user experience and optimize system performance. The code_execution_tool.py has been significantly upgraded to manage extensive terminal tasks more effectively, with new features like prompt handling, idle timeout detection, and summarized output for efficient API communication. Additionally, new setup instructions and configuration changes have been implemented across various files to support these updates. See README for in-depth setup instructions. These changes collectively aim to provide a more robust and user-friendly environment, ensuring better resource management and smoother operations.

Summary of Changes

agent.py
-Updated default docker_name and docker_image configurations.

build.txt
-Modified entries to reflect changes in docker_name and docker_image.

code_execution_tool.py
-Implemented elaborate updates for improved prompt handling and response pass-back.
-Added idle timeout detection, ensuring the agent doesn't spam identical terminal commands to the user.
-Introduced a summarized output feature to manage API token usage during extensive terminal tasks.
-Enhanced handling to retain terminal control, with agent check-ins only occurring upon idle detection (approximately 30 seconds).

docker.py
-Increased sleep duration to accommodate potentially longer container spin-up times.

docker-compose.yml
-Added for creating the NVIDIA image. Detailed instructions can be found in the README.

Dockerfile
-Adjusted to include a new container setup (previous setup commented out).
-Added automatic timezone detection for image composition.
-General reorganization and cleanup.

main.py
-Clarified commented-out docker_name references for better readability.

rate_limiter.py
-Fixed a bug affecting the agent during long tasks and numerous calls.

README.md
-Added a comprehensive section for Ubuntu and NVIDIA CUDA runtime setup and troubleshooting.

timeout_input.py
-Reverted to its original value.

@deciduus
Copy link
Author

deciduus commented Aug 4, 2024

KNOWN ISSUE - sometimes on an incomplete terminal output the agent will no check in as quickly as it would on the original - however, the idle count/score will kick in and pass it back to user after approximately 30 seconds.

If however it gets stuck and you want to manually pass back, simply user intervention, prompt, and or pass it details about how it was stuck, or looping, etc.

TLDR if you get stuck, wait, or E and relaunch 'python main.py'

…plified)

extensive dockerfile changes to remove venv and add custom package installs when composing docker image. (machine learning packages)
adjusted the image reference comment on main for my fork
readme is a bit of a mess (not gracefully formatted additions) but i added some troubleshooting info for docker/ubuntu composing, images etc.
@deciduus
Copy link
Author

deciduus commented Aug 7, 2024

A lot of these changes are specific to my fork, but we should be able to add optionality. I haven't focused on this aspect as much, as I've been primarily building new functionality.

  • Bash Script Simplification: Edited the bash script to remove the virtual environment setup. Given that the application runs inside a Docker container, the virtual environment was deemed unnecessary.
  • Dockerfile Modifications: Made extensive changes to the Dockerfile to:
    • Eliminate the virtual environment setup.
    • Included the installation of custom machine learning packages during the Docker image build process. Currently, this requires manual composition and runtime verification. Note that upon first system launch or after a system restart, the NVIDIA Docker setup may encounter issues where the runtime (daemon) reverts to the default Docker settings. This necessitates a manual override of the daemon configuration. Detailed troubleshooting steps for this process are provided in the README.
  • Image Reference Update: Adjusted the image reference comment in the main code to reflect the updated image name.
  • README Updates: Added troubleshooting information related to Docker and Ubuntu image composition and other relevant details. Note that the additions are functional but not yet gracefully formatted.

These changes streamline the setup process for nvidia cuda and packages (custom image) and improve the overall functionality of the application within a Docker environment.

@deciduus
Copy link
Author

deciduus commented Aug 7, 2024

I've resolved conflicts -
-Reformatted and optimized the agent.system.md to incorporate the updates from frdel:main, as well as incorporate smaller elements from my previous edits within that context.
-Integrated the iteration logic from the fork for checking existing containers, ensuring robustness in finding containers by name, and gracefully closing the container on exit (personal preference)
-Retained PrintStyle import from the main branch and included Wikipedia search functionality from the fork.

Comment on lines +60 to +80
except NotFound:
try:
print(f"Creating new container: {self.name}")
self.container = self.client.containers.run(
self.image,
detach=True,
ports=self.ports,
name=self.name,
volumes=self.volumes,
)
print(f"Started new container with ID: {self.container.id}")
except docker.errors.ImageNotFound as inf:
print(f"Image not found: {inf}")
except docker.errors.APIError as api_error:
print(f"Docker API error: {api_error}")
except Exception as e:
print(f"Unexpected error when starting container: {e}")
finally:
atexit.register(self.cleanup_container)
time.sleep(5)
self.start_health_check_thread()

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd opt to break out to variable instead for readability, just like in the original

@deciduus
Copy link
Author

deciduus commented Aug 8, 2024 via email

@TitanKing
Copy link

TitanKing commented Aug 16, 2024

This code causes a error loop, I am not sure if it is the renaming of the docker file I will have to test, but it sends any ai into a loop with:

Code: touch test.txt

Creating new container: docker-agent-zero-exe
Image not found: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=latest&fromImage=docker-agent-zero-exe: Not Found ("pull access denied for docker-agent-zero-exe, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")
SSH Connection attempt 1...
SSH Connection attempt 2...

{
    "system_error": "Traceback (most recent call last):
  File "/Users/jasonschoeman/Documents/Projects/agent-zero-2/python/helpers/shell_ssh.py", line 27, in connect
    self.client.connect(self.hostname, self.port, self.username, self.password)
  File "/Users/jasonschoeman/.pyenv/versions/3.12.5/lib/python3.12/site-packages/paramiko/client.py", line 409, in connect
    raise NoValidConnectionsError(errors)
paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 50022 on 127.0.0.1 or ::1
"
}

@deciduus
Copy link
Author

deciduus commented Aug 16, 2024 via email

…ide (for redundant reference)

code execution tool summarizing extended for more context on longer files.
@deciduus
Copy link
Author

updated the nvidia cuda setup steps to address errors and prevent the docker desktop overwriting the default runtime.

@InfantLab
Copy link

I tested this as a clean install. And it seemed to work nicel y for me (Win11, CUDA 12.4, GeForce GTX 980Ti ). Thanks, deciduus.

DanozWorld added a commit to DanozWorld/tazeeka-fusion that referenced this pull request Aug 19, 2024
Incorporate changes from deciduus agent-zero fork [~ IDE terminal compatibility, docker quality of life, prompt optimizations, execution handling & style encoding, NVIDIA CUDA (GPU support) ~ frdel#19]
@deciduus
Copy link
Author

deciduus commented Aug 19, 2024 via email

DanozWorld added a commit to DanozWorld/tazeeka-fusion that referenced this pull request Aug 19, 2024
Deciduus agent zero fork changes (Incorporate changes from deciduus agent-zero fork [~ IDE terminal compatibility, docker quality of life, prompt optimizations, execution handling & style encoding, NVIDIA CUDA (GPU support) ~ frdel#19]
@TitanKing
Copy link

If you had the container running from composing it up you need to close it prior to launch in this scenario.

On Fri, Aug 16, 2024, 9:14 AM Jason Schoeman @.> wrote: This code causes a crash, I am not sure if it is the renaming of the docker file but I will have to test, but it sends any ai into a loop with: Code: touch test.txt Creating new container: docker-agent-zero-exe Image not found: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=latest&fromImage=docker-agent-zero-exe: Not Found ("pull access denied for docker-agent-zero-exe, repository does not exist or may require 'docker login': denied: requested access to the resource is denied") SSH Connection attempt 1... SSH Connection attempt 2... { "system_error": "Traceback (most recent call last): File "/Users/jasonschoeman/Documents/Projects/agent-zero-2/python/helpers/shell_ssh.py", line 27, in connect self.client.connect(self.hostname, self.port, self.username, self.password) File "/Users/jasonschoeman/.pyenv/versions/3.12.5/lib/python3.12/site-packages/paramiko/client.py", line 409, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 50022 on 127.0.0.1 or ::1 " } — Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BEA7AQ7E3XJB4USYO5PVDOLZRYCMDAVCNFSM6AAAAABLWPMCDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJTGU4TAMJXGY . You are receiving this because you authored the thread.Message ID: @.>

I deleted the container completely, it is on a MacOs, still get the same issue:
Image not found: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=latest&fromImage=docker-agent-zero-exe: Not Found ("pull access denied for docker-agent-zero-exe, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

I dont get it on the main branch.

@deciduus
Copy link
Author

deciduus commented Aug 19, 2024 via email

@TitanKing
Copy link

You may need to purge all the old images/containers in ubuntu, recompose the image, and try again. I've also updated the README and setup instructions since your original post, it may be worth trying again from step 1. On Mon, Aug 19, 2024 at 10:24 AM Jason Schoeman @.> wrote:

If you had the container running from composing it up you need to close it prior to launch in this scenario. … <#m_8730107855366028201_> On Fri, Aug 16, 2024, 9:14 AM Jason Schoeman @.
> wrote: This code causes a crash, I am not sure if it is the renaming of the docker file but I will have to test, but it sends any ai into a loop with: Code: touch test.txt Creating new container: docker-agent-zero-exe Image not found: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=latest&fromImage=docker-agent-zero-exe: Not Found ("pull access denied for docker-agent-zero-exe, repository does not exist or may require 'docker login': denied: requested access to the resource is denied") SSH Connection attempt 1... SSH Connection attempt 2... { "system_error": "Traceback (most recent call last): File "/Users/jasonschoeman/Documents/Projects/agent-zero-2/python/helpers/shell_ssh.py", line 27, in connect self.client.connect(self.hostname, self.port, self.username, self.password) File "/Users/jasonschoeman/.pyenv/versions/3.12.5/lib/python3.12/site-packages/paramiko/client.py", line 409, in connect raise NoValidConnectionsError(errors) paramiko.ssh_exception.NoValidConnectionsError: [Errno None] Unable to connect to port 50022 on 127.0.0.1 or ::1 " } — Reply to this email directly, view it on GitHub <#19 (comment) <#19 (comment)>>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BEA7AQ7E3XJB4USYO5PVDOLZRYCMDAVCNFSM6AAAAABLWPMCDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJTGU4TAMJXGY https://github.com/notifications/unsubscribe-auth/BEA7AQ7E3XJB4USYO5PVDOLZRYCMDAVCNFSM6AAAAABLWPMCDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJTGU4TAMJXGY . You are receiving this because you authored the thread.Message ID: @.
> I deleted the container completely, it is on a MacOs, still get the same issue: Image not found: 404 Client Error for http+docker://localhost/v1.46/images/create?tag=latest&fromImage=docker-agent-zero-exe: Not Found ("pull access denied for docker-agent-zero-exe, repository does not exist or may require 'docker login': denied: requested access to the resource is denied") I dont get it on the main branch. — Reply to this email directly, view it on GitHub <#19 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/BEA7AQ5KM7TJ2UODJCYPJ3DZSIE4HAVCNFSM6AAAAABLWPMCDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJWHA2DKOBZGE . You are receiving this because you authored the thread.Message ID: @.
**>

Thanks for your response, I got it working by overriding these in the main.py:

        code_exec_docker_enabled = True,
        code_exec_docker_name = "agent-zero-exe",
        code_exec_docker_image = "frdel/agent-zero-exe:latest",
        code_exec_docker_ports = {"22/tcp": 50022},
        code_exec_docker_volumes = {files.get_abs_path("work_dir"): {"bind": "/root", "mode": "rw"}},
        code_exec_ssh_enabled = True,
        code_exec_ssh_addr = "localhost",
        code_exec_ssh_port = 50022,
        code_exec_ssh_user = "root",
        code_exec_ssh_pass = "toor",

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants