Evaluate groq

[cmungall]

groq has jawdroppingly fast access to mixtral. Currently you can use the UI and API for no cost. There is throttling but it seems quite generous

it's easy to use via the awesome litellm

See https://github.com/monarch-initiative/curate-gpt/blob/main/README.md#selecting-models for general setup

First make sure you are up to date

pipx update litellm

then fire it up:

litellm -m groq/mixtral-8x7b-32768

Add this to extra-openai-models.yaml as detailed in the llm docs:

- model_name: litellm-groq-mixtral
  model_id: litellm-groq-mixtral
  api_base: "http://0.0.0.0:8000"

You can use the CLI: llm -m litellm-groq-mixtral "10 names for a pet pelican"

[cmungall]

I've been using this in OAK. It seems less responsive than it was. Overall mixtral works well but will frequently escape underscores in JSON keys as if it were a latex document...