Litellm dev 12 26 2024 p3 (#7434)

* build(model_prices_and_context_window.json): update groq models to specify 'supports_vision' parameter Closes #7433 * docs(groq.md): add groq vision example to docs Closes #7433 * fix(prometheus.py): refactor self.litellm_proxy_failed_requests_metric to use label factory * feat(prometheus.py): new 'litellm_proxy_failed_requests_by_tag_metric' allows tracking failed requests by tag on proxy * fix(prometheus.py): fix exception logging * feat(prometheus.py): add new 'litellm_request_total_latency_by_tag_metric' enables tracking latency by use-case * feat(prometheus.py): add new llm api latency by tag metric * feat(prometheus.py): new litellm_deployment_latency_per_output_token_by_tag metric allows tracking deployment latency by tag * fix(prometheus.py): refactor 'litellm_requests_metric' to use enum values + label factory * feat(prometheus.py): new litellm_proxy_total_requests_by_tag metric allows tracking total requests by tag * feat(prometheus.py): new metric litellm_deployment_successful_fallbacks_by_tag allows tracking deployment fallbacks by tag * fix(prometheus.py): new 'litellm_deployment_failed_fallbacks_by_tag' metric allows tracking failed fallbacks on deployment by custom tag * test: fix test * test: rename test to run earlier * test: skip flaky test
BerriAI · Dec 27, 2024 · 9d82ff4 · 9d82ff4
1 parent 17d5ff2
commit 9d82ff4
Show file tree

Hide file tree

Showing 7 changed files with 389 additions and 78 deletions.
diff --git a/docs/my-website/docs/providers/groq.md b/docs/my-website/docs/providers/groq.md
@@ -259,6 +259,99 @@ if tool_calls:
     print("second response\n", second_response)
 ```
 
+## Groq - Vision Example    
+
+Select Groq models support vision. Check out their [model list](https://console.groq.com/docs/vision) for more details.
+
+<Tabs>
+<TabItem value="sdk" label="SDK">
+
+```python
+from litellm import completion
+
+import os 
+from litellm import completion
+
+os.environ["GROQ_API_KEY"] = "your-api-key"
+
+# openai call
+response = completion(
+    model = "groq/llama-3.2-11b-vision-preview", 
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                            {
+                                "type": "text",
+                                "text": "What’s in this image?"
+                            },
+                            {
+                                "type": "image_url",
+                                "image_url": {
+                                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
+                                }
+                            }
+                        ]
+        }
+    ],
+)
+
+```
+
+</TabItem>
+<TabItem value="proxy" label="PROXY">
+
+1. Add Groq models to config.yaml   
+
+```yaml
+model_list:
+  - model_name: groq-llama3-8b-8192 # Model Alias to use for requests
+    litellm_params:
+      model: groq/llama3-8b-8192
+      api_key: "os.environ/GROQ_API_KEY" # ensure you have `GROQ_API_KEY` in your .env
+```
+
+2. Start Proxy
+
+```bash
+litellm --config config.yaml
+```
+
+3. Test it
+
+```python
+import os 
+from openai import OpenAI
+
+client = OpenAI(
+    api_key="sk-1234", # your litellm proxy api key
+)
+
+response = client.chat.completions.create(
+    model = "gpt-4-vision-preview",  # use model="llava-hf" to test your custom OpenAI endpoint
+    messages=[
+        {
+            "role": "user",
+            "content": [
+                            {
+                                "type": "text",
+                                "text": "What’s in this image?"
+                            },
+                            {
+                                "type": "image_url",
+                                "image_url": {
+                                "url": "https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg"
+                                }
+                            }
+                        ]
+        }
+    ],
+)
+
+```
+</TabItem>
+</Tabs>
+
 ## Speech to Text - Whisper
 
 ```python
@@ -274,4 +367,5 @@ transcript = litellm.transcription(
 )
 
 print("response=", transcript)
-```
+```
+