Experiments for the paper

[imenelydiaker]

Since the deadline is approaching, I'm not sure we'll be able to run the whole benchmark. I suggest the following list of experiments, that imo is enough for the paper.

Suggested experiments for the paper

• Run the English benchmark, with and without Fast clustering versions.
• Run a new benchmark for another language, German or Russian Add Russian tasks (RU-MTEB) #815 ? (Chinese, French and Scandinavian are already included in their respective papers).
• Gather all newly added datasets results (e5-mutlilingual-small and paraphrase-multilingual-all-MiniLM-12-v2) and add them to a table in the appendix.
• Ideally, if we have enough datasets, it would be great to have a benchmark on one/many low-ressource languages. Or just show results on some low-ressource languages on one task type (e.g., classification).

Expected outputs:

• Table of English benchmark before/after Fast versions.
• Table of German benchmark or any other language.
• Table with results of all newly added tasks.
• Need some nice figures, like the distribution of languages per task type. Or speedups due to the Fast version of clustering tasks.
• Report carbon emissions of the English benchmark in Ethical considerations section.

Models

From discussion #705 (comment)

• all-MiniLM-*-v2 [these are English models]
• paraphrase-multilingual-*
• all-mpnet-*
• multilingual-e5-*
• e5-mistral-7b-instruct (as it was the first to use Mistral+instructions for embedding)
• voyage-large-2-instruct (as it is first on the MTEB leaderboard right now 1 week ago)
• text-embedding-large-3 (due to its relevance in the industry)
• GritLM-7b & LLM2Vec
• LASER
• nomic-embed-text-v1
• BAAI/bge-m3
• Add popular Cross-Encoders (since we support them now)

These are just suggestions. Feel free to edit or suggest other experiments.

[mrshu]

@imenelydiaker can we order the models in order of priority? I can start running them across the whole benchmark now.

[isaac-chung]

@imenelydiaker would you mind linking the respective french/chinese benchmark papers as well? might be useful for paper writing and how to structure our analyses

[imenelydiaker]

Chinese MTEB: https://arxiv.org/abs/2309.07597
French paper: https://arxiv.org/abs/2405.20468 (an updated version will be uploaded to arxiv soon)