MTEB is a benchmark that spans 8 embedding tasks covering a total of 56 datasets and 112 languages. The 8 task types are Bitext mining, Classification, Clustering, Pair Classification, Reranking, Retrieval, Semantic Textual Similarity and Summarisation. The 56 datasets contain varying text lengths and they are grouped into three categories: Sentence to sentence, Paragraph to paragraph, and Sentence to paragraph.

Check the latest leaderboards at HuggingFace.

AlphaNLI An MTEB dataset Massive Text Embedding Benchmark

Measuring the ability to retrieve the groundtruth answers to reasoning task queries on AlphaNLI.

Task category t2t

Domains Encyclopaedic, Written

Reference https://leaderboard.allenai.org/anli/submissions/get-started

  How to evaluate on this task

You can evaluate an embedding model on this dataset using the following code: import mteb

task = mteb.get_task("AlphaNLI") evaluator =… See the full description on the dataset page: https://huggingface.co/datasets/mteb/AlphaNLI.

SIQA An MTEB dataset Massive Text Embedding Benchmark

Measuring the ability to retrieve the groundtruth answers to reasoning task queries on SIQA.

Task category t2t

Domains Encyclopaedic, Written

Reference https://leaderboard.allenai.org/socialiqa/submissions/get-started

  How to evaluate on this task

You can evaluate an embedding model on this dataset using the following code: import mteb

task = mteb.get_task("SIQA") evaluator = mteb.MTEB([task])… See the full description on the dataset page: https://huggingface.co/datasets/mteb/SIQA.

Leadboard

BCEmbedding: Bilingual and Crosslingual Embedding for RAG

GitHub

Click to Open Contents

🌐 Bilingual and Crosslingual Superiority 💡 Key Features 🚀 Latest Updates 🍎 Model List 📖 Manual Installation Quick Start

⚙️ Evaluation Evaluate Semantic Representation by MTEB Evaluate RAG by LlamaIndex

📈 Leaderboard Semantic Representation Evaluations in MTEB RAG Evaluations in LlamaIndex

🛠 Youdao's BCEmbedding API 🧲 WeChat Group ✏️ Citation 🔐… See the full description on the dataset page: https://huggingface.co/datasets/maidalun1020/CrosslingualMultiDomainsDataset.