[Feature]- Support for the microsoft/Phi-3-vision-128k-instruct Vision Model

[sabarish244]

Motivation

The latest release of microsoft phi3 4.2b 128k context vision model looks promising in performance and resource saving one too as it boast just 4.2b parameter. So it would be a great feature if the lmdeploy inference server supports it

Related resources

https://huggingface.co/microsoft/Phi-3-vision-128k-instruct/tree/main
https://azure.microsoft.com/en-us/blog/new-models-added-to-the-phi-3-family-available-on-microsoft-azure/

Additional context

I tried running the model via the lmdeploy docker inference server, installed the required additional packages and ran the model, model loaded and running but while trying to inference it via the api's we are getting either empty response or internal server error

[RunningLeon]

@sabarish244 Hi, thanks for your info. We include this model in the TODO list.

[Youho99]

A first implementation on the vLLM project was made of Phi3-Vision.

Maybe this can help.

vllm-project/vllm#4986

[RunningLeon]

@Youho99 hi, welcome to try this PR #1845 .