Running models

[imenelydiaker]

Should we start running models for the paper?

We can define the scope here:

1. Number of models to run and their type (small, large ones, should also consider closed-source models).
2. Any idea of how to aggregate more than 443 tasks results in a nice readable table/figure? (open question on how to show the results)
3. Are we keeping mteb/results? The only objection I see for merging it within the current repo is that it could lead to errors: contributors may delete or update files unintentionally and we may miss it during reviews as it gets lost between code files.
4. Where should we put the scripts that we'll use to run experiments?
5. Following the discussion here Tracking CO2 Emissions #703 : it would be really nice if we could estimate the carbon footprint of our experiments.

Any other considerations before starting the experiments?
@Muennighoff @KennethEnevoldsen @isaac-chung @x-tabdeveloping @orionw

[x-tabdeveloping]

Some models we should probably have:

• all-MiniLM-*-v2
• paraphrase-multilingual-*
• all-mpnet-*
• multilingual-e5-*
• e5-mistral-7b-instruct (as it was the first to use Mistral+instructions for embedding)
• voyage-large-2-instruct (as it is first on the MTEB leaderboard right now)
• text-embedding-large-3 (due to its relevance in the industry)
• GritLM-7b & LLM2Vec supervised and unsupervised (due to methodological novelty and usefulness for creating new models)

Anything else comes to mind?

[KennethEnevoldsen]

I would add LASER to the list as well.

[wissam-sib]

Here are some other models I can think of:

• https://huggingface.co/sentence-transformers/LaBSE
• sentence-transformers/distiluse-base-multilingual-*
• xlm-roberta-base-multilingual-*
• https://huggingface.co/nomic-ai/nomic-embed-text-v1
• BAAI/bge-m3

Would be nice to also run one of the best english models (and a few others monolingual models for the most common languages).

[bp-high]

How many maximum models are gonna be run btw? Also should models trained on a family on languages (Indic languages, European languages) be tested to see how well they perform on the whole benchmark and the languages they were trained on?

A few language family specific models that come to my mind:

• l3cube-pune/indic-sentence-similarity-sbert (Indic Languages)
• symanto/sn-xlm-roberta-base-snli-mnli-anli-xnli (European + Asian languages)
• clips/mfaq (European languages)

[Ab1992ao]

Hi there, we are the team who create russian MTEB part. Our opinion mono lang models are important for evaluation, because they metrics often have higher scores that multiling models. Our team takes a models aboard to eval on ru part of mteb:

ai-forever/sbert_large_nlu_ru
ai-forever/sbert_large_mt_nlu_ru
cointegrated/rubert-tiny
cointegrated/rubert-tiny2

We are reserve it for ours team, thanks.

[KennethEnevoldsen]

Sounds wonderful @Ab1992ao

[KranthiGV]

Can volunteers help with running the models?

[Ruqyai]

I would like to help and if there are steps that we follow, that would be great

[mrshu]

This will sound crazy, but the benchmark has been running for a week on a single H100 with sentence-t5-xxl and still has not finished. Having inspected the results closely, it seems we are in the middle of the Retrieval tasks right now, where each one takes about an hour. If all goes well, we should be done tomorrow.

[KennethEnevoldsen]

thanks @mrshu this indeed seems to be problematic. When you share the scores we should be able to inspect where the most promising speed-ups are.

[mrshu]

Just so that I don't forget, here are the results for the sentence-t5-xxl model: some of them are CO2 numbers are indeed crazy large: main...mrshu:mteb:mrshu/port-carbon-emissions-estimation

[mrshu]

Just a small ping here -- happy to run some more experiments if it would help!