[R-259] Which is the best LLM for evaluation?

[yadavshashank]

I checked the documentation and related resources and couldn't find an answer to my question.

Your Question
Do RAGAS prompts work equally well with other LLMs like Claude 3 Sonnet and Llama 3? If not which model to choose?
Also, is there a way to print and modify the prompts?

Additional context

• I can see a big variation in scores across different models.
• GPT-3.5 gives me higher scores compared to others for all metrics. Claude 3 Sonnet is like the average of Llama 3 and GPT-4 Turbo. Llama 3 and Cohere Command often give NaN output for some metrics.
• My evaluation set has 19 records.

Radar chart comparison of scores:

_R-259

[shahules786]

Hey @yadavshashank This is a very interesting analysis. This is also a problem we have been thinking about. Fundamentally what matters here is which of these LLMs would suit your level of alignment between automated scoring vs manual scoring. This is the problem we can solve by adding some form of UI component before automated scoring that would allow developers to do some level of manual checking and make sure that scores align with their judgment. We will be working in this direction, but that will only come later.
As of today, my best recommendation is to go for the best overall performing LLMs like gpt-4 / Claude if closed source and llama-3 if opensource.

Would love to hop on a call and help/chat with you if you're open. My cal is here.