-
Notifications
You must be signed in to change notification settings - Fork 202
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sampling MTEB #647
Comments
Not at the moment @NehaB18, we are however working on speeding the benchmark up and have already had a few drastic improvements (#572, #481). For retrieval there is an ongoing discussion over at #638 Implementing a downsampling function for the retrieval might be a reasonable solution to speeding it up. If you simply want to run it on a selected subset of the retrieval task you can do something like: import mteb
import random
tasks = mteb.get_tasks(languages = ["eng"], domains = ["Legal"], task_types = ["Retrieval"])
task_list = [t for t in tasks]
random.shuffle(task_list)
tasks_to_run = tasks[:10] # select the 10 first tasks |
@NehaB18 I will move this over to the discussions |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
is there any way to run the evaluation on the sample of datasets for example 5% of all Retrieval tasks?
The text was updated successfully, but these errors were encountered: