Crash when uploading an image with model Llama 3 70B

[Hugobox]

Bug Report

Description

Bug Summary:
We use model Llama 3 70B via litellm and it works well, but as soon as we upload any image file (png or jpg) the webui becomes unresponsive and has to be restarted, no error found in logs. PDFs do not crash webui, only images.

Normally with models that do not support image uploads like GPT3.5-turbo, we get the following error message:
"Uh-oh! There was an issue connecting to gpt-3.5-turbo.
External: Invalid content type. image_url is only supported by certain models."

But it seems like with Llama 3 70B via litellm this file type check is not done correctly.

Steps to Reproduce:
use model Llama 3 70B via litellm
open webui, set model to Llama 3 70B
test model with text prompt, notice it works
test model with image upload and text prompt, notice it becomes unresponsive.

Expected Behavior:
[Describe what you expected to happen.]

Actual Behavior:
[Describe what actually happened.]

Environment

• Open WebUI Version: v0.1.125

• Operating System: [Ubuntu 20.04]

Reproduction Details

Confirmation:

• [ x] I have read and followed all the instructions provided in the README.md.
• [ x] I am on the latest version of both Open WebUI and Ollama.
• I have included the browser console logs.
• I have included the Docker container logs.

Logs and Screenshots

Browser Console Logs:
[Include relevant browser console logs, if applicable]

Docker Container Logs:
[Include relevant Docker container logs, if applicable]

Screenshots (if applicable):
[Attach any relevant screenshots to help illustrate the issue]

Installation Method

Docker-compose file

Additional Information

[Include any additional details that may help in understanding and reproducing the issue. This could include specific configurations, error messages, or anything else relevant to the bug.]

Note

If the bug report is incomplete or does not follow the provided instructions, it may not be addressed. Please ensure that you have followed the steps outlined in the README.md and troubleshooting.md documents, and provide all necessary information for us to reproduce and address the issue. Thank you!

[justinh-rahb]

Llama3 is not multi-modal, it does not support images as input.

[tjbck]

Built-in LiteLLM will be deprecated in the near future.

[Hugobox]

@justinh-rahb Yes but the webui doesn't know that and allows images to be uploaded and hangs.
@tjbck Ok, what will replace litellm?

[tjbck]

It just won't be bundled with open-webui anymore, and you should still be able to add self-hosted LiteLLM instance via Open WebUI OpenAI APIs.

[justinh-rahb]

@justinh-rahb Yes but the webui doesn't know that and allows images to be uploaded and hangs. @tjbck Ok, what will replace litellm?

There's an ongoing effort to add more metadata for models, including a flag for whether they support multi-modal input.

[Hugobox]

@tjbck Ok I understand better now, but we are already using a self-hosted liteLLM instance so that won't be a problem for us.
@justinh-rahb Where would the flag be set? In webUI, liteLLM? Thanks!

[spammenotinoz]

It just won't be bundled with open-webui anymore, and you should still be able to add self-hosted LiteLLM instance via Open WebUI OpenAI APIs.

You can use a Self Hosted deployment of LiteLLM now, update the OpenAI config including BASE URL to point to your LiteLLM deployment.
LiteLLM has a mapping of which models support images, but it's also up to the front-end, so intrigued how Open-Web will handle this.
Anthropic has terrible handling of images, expects them to be 400kb, but technically handles up to 4mb. Was thinking of butchering the code when an Image Embedding is used force a model like GPT-4o that actually seems reliable with images.

[Hugobox]

After further testing, its not so much a crash but more a freeze as docker doesn't restart the conatiner even if in "always-restart" mode, so we now have a docker healthcheck in place with docker autoheal to restart the container to reduce downtime.

[spammenotinoz]

Same occurs if you change to a non-vision model if an image exists anywhere in the convert history.
Try the Dev branch, appears to handle a little better,
You can disable images via models,
So less freezes, just spews “model does not support images” if there is an image in the chat (user submitted or otherwise”

[Hugobox]

After upgrading to openWebui v.0.2.4, if we try to upload an image while using the same model (Llama 3) via vLMM (not liteLLM, did not try an external liteLLM instance yet), openWebUI doesn't freeze anymore.

[spammenotinoz]

Yes, it has improved flashes a RED Error though about this model does not support vision.

[justinh-rahb]

considering llama3 does not in fact support images... seems to be working as intended now. Previous behaviour where it would crash after trying anyway was a bug.

[spammenotinoz]

While true, it's inconsistent behaviour with other platforms so can be confusing. On one hand it's great to continue a conversation with another model, but 99% of the time when I change models, it will be a new context, so carrying across the context just creates the potential for bugs like this. If changing models results in an error (no longer freeze) for every subsequent message in the conversation, then surely that is a bug.

…

________________________________ From: Justin Hayes ***@***.***> Sent: Thursday, 6 June 2024 8:18 AM To: open-webui/open-webui ***@***.***> Cc: spammenotinoz ***@***.***>; Comment ***@***.***> Subject: Re: [open-webui/open-webui] Crash when uploading an image with model Llama 3 70B (Discussion #2498) considering llama3 does not in fact support images... seems to be working as intended now. Previous behaviour where it would crash after trying anyway was a bug. — Reply to this email directly, view it on GitHub<#2498 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AG7KS4AWHAEQVBMBDNJRIULZF6FE5AVCNFSM6AAAAABIEA55Z6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TMOBSHEZTE>. You are receiving this because you commented.Message ID: ***@***.***>

[justinh-rahb]

Both things can be true. My first pass at ideating a solution looks something like:

1. Conversations have a state flag for whether there is currently a multimodal-capable model in use.
2. If true, image payloads in the conversation context are passed to the API unmodified. If false we filter them out before sending a generation request.
3. Flipping the state does not need to modify the conversation history or database record, it simply needs to control what is sent to the backend.

[spammenotinoz]

Curious do you actually switch models and continue the conversation with a new model? perhaps I should try this more, still quite new to testing this project.
Rather than code for exceptions, thinking it may be cleaner to align with ChatGPT and other projects to just clear the history when changing models at least when selecting a model from the pull-down menu.
The one issue that has stopped me switching, is when using \ testing under this project API calls are 10x + higher. There is limited prompt\compression\smarts around conversation history and seems different handling between Ollama vs OpenAI models.
Guess it's intended more for power users or for purists, which is great, but starting to grow on me, the team have done an excellent job with the interface and models. The models feature is working with OpenAI and is blowing my mind.