[v2] Remove MultilingualTask

[Samoed]

I think we can integrate the MultilingualTask class into the AbsTask class. The most useful function is already there, and it can be easily integrated into load_data.

mteb/mteb/abstasks/MultilingualTask.py

Lines 56 to 61 in ce5cb3e

	self.dataset = {}
	for lang in self.hf_subsets:
	self.dataset[lang] = datasets.load_dataset(
	name=lang,
	**self.metadata.dataset,
	)

Additionally, we should reupload datasets with fast_loading, as using parquet files will make them much faster.

mteb/mteb/abstasks/MultilingualTask.py

Lines 39 to 52 in ce5cb3e

	self.dataset = {}
	merged_dataset = datasets.load_dataset(
	**self.metadata.dataset
	) # load "default" subset
	for split in merged_dataset.keys():
	df_split = merged_dataset[split].to_polars()
	df_grouped = dict(df_split.group_by(["lang"]))
	for lang in set(df_split["lang"].unique()) & set(self.hf_subsets):
	self.dataset.setdefault(lang, {})
	self.dataset[lang][split] = datasets.Dataset.from_polars(
	df_grouped[(lang,)].drop("lang")
	) # Remove lang column and convert back to HF datasets, not strictly necessary but better for compatibility
	for lang, subset in self.dataset.items():
	self.dataset[lang] = datasets.DatasetDict(subset)