Introducing MTEB v2: Multimodal Embedding Evaluation

On October 20, 2025 Hugging Face released **MTEB v2**, a significant refactoring of the Massive Text Embedding Benchmark, which was originally designed for evaluating text embedding models across various tasks like classification and retrieval. The update addresses **package bloating and the need for broader support** by introducing a **more consistent interface, better typing, and improved documentation**. Key new features include support for **multimodal evaluation (text, images, and audio)**, **unified retrieval and reranking tasks**, and an **easier evaluation process** using the new `mteb.evaluate` function and `ResultCache` for managing results. The article also provides detailed instructions for **upgrading from MTEB v1**, including how to convert old models and datasets to the new v2 format.

Source:

https://huggingface.co/blog/isaacchung/mteb-v2