Integrate JMTEB

[Muennighoff]

Would be great to have the datasets in JMTEB (https://github.com/sbintuitions/JMTEB) integrated into MTEB for those which aren't yet already, so we can also add a Japanese leaderboard sometime 😊 cc @lsz05 @ryokan0123 @masaya-ohagi & anyone who may be interested

[awinml]

I would like to help integrate JMTEB. The following datasets from JMTEB are already present in MTEB:

• AmazonCounterfactualClassification
• AmazonReviewClassification
• MassiveIntentClassification
• MassiveScenarioClassification
• JSTS
• PawsX PairClassification

The following ones would need to be added:

• Clustering: (fix: Add JMTEB Clustering datasets #768)

  • Livedoor News
  • MewsC-16-ja

• STS: (fix: Add JSICK dataset #769)

  • JSICK

• Retrieval: (fix: Add JaGovFaqs and NLPJournal datasets #808)

  • NLP Journal title-abs
  • NLP Journal title-intro
  • NLP Journal abs-intro
  • JaGovFaqs-22k
  • JAQKET
  • Mr.TyDi-ja

• Reranking:

  • ESCI

[AlexeyVatolin]

@Muennighoff, I can take over the integration of ESCI. I have already looked at the contents of this dataset. In the original dataset https://huggingface.co/datasets/tasksource/esci there are three languages: English, Spanish and Japanese. Do we want to add all three languages or just Japanese?

[Muennighoff]

@Muennighoff, I can take over the integration of ESCI. I have already looked at the contents of this dataset. In the original dataset https://huggingface.co/datasets/tasksource/esci there are three languages: English, Spanish and Japanese. Do we want to add all three languages or just Japanese?

I think if we can add all 3 that'd be even better! 🙌