TLDR: VN-MTEB is a new, large-scale benchmark for evaluating text embedding models in Vietnamese. It comprises 41 datasets across six NLP tasks, created by automatically translating English datasets using LLMs and advanced filtering techniques to ensure high quality. The benchmark helps researchers assess model performance for Vietnamese, revealing that larger, RoPE-based, and instruct-tuned models generally perform better.
The landscape of Natural Language Processing (NLP) has been significantly shaped by the rapid advancements in Large Language Models (LLMs). However, a notable disparity exists in the availability of comprehensive evaluation benchmarks across different languages. While widely spoken languages like English and Chinese benefit from extensive resources, low-resource languages, such as Vietnamese, have historically lacked large-scale, diverse benchmarks for evaluating AI models.
This critical gap is now being addressed with the introduction of VN-MTEB, the Vietnamese Massive Text Embedding Benchmark. This new benchmark is designed to provide a standardized and robust framework for assessing the performance of text embedding models specifically for the Vietnamese language. Text embeddings are fundamental to many modern applications, including recommendation systems and content moderation, which are particularly vital in a country like Vietnam, known for its high internet usage and the associated challenges of online content.
The creation of VN-MTEB involved a sophisticated process of translating a vast number of English samples from the established Massive Text Embedding Benchmark (MTEB). The researchers developed an innovative automated framework that harnesses the capabilities of cutting-edge LLMs and embedding models. This framework ensures that the translated datasets maintain a high level of quality, preserving the natural flow of language, semantic accuracy, and crucial elements like named entities and code snippets.
VN-MTEB is a truly comprehensive benchmark, encompassing 41 datasets categorized under six distinct NLP tasks: retrieval, reranking, classification, clustering, pair classification, and semantic textual similarity. This broad coverage allows for a much more thorough and nuanced evaluation of Vietnamese text embedding models compared to previous, more limited approaches.
The automated translation pipeline is a cornerstone of this research. It operates in three main stages: initially, it filters for samples in the desired source language; subsequently, it translates these datasets using an LLM; and finally, it rigorously evaluates the generated translations. This evaluation stage is crucial, employing checks for language contamination, semantic similarity against the original content, and an overall quality score based on criteria such as grammar, named entity recognition (NER), fluency, and meaning preservation. This multi-layered validation significantly reduces the need for manual human intervention, making the process highly scalable and efficient for large datasets.
In their analysis, the researchers observed that larger and more intricate models, particularly those that incorporate Rotary Positional Embedding (RoPE), generally demonstrated superior performance over models utilizing Absolute Positional Embedding (APE) in various embedding tasks. Furthermore, instruct-tuned models, which are specifically trained with natural language descriptions of tasks, consistently showed enhanced overall performance. The study evaluated 18 different embedding models, offering valuable insights into their strengths and weaknesses across the diverse tasks within the VN-MTEB framework.
Also Read:
- A New Milestone in Vietnamese Legal AI: Introducing the VLQA Dataset
- A New Standard for Evaluating AI’s Grasp of Government Affairs
This work not only fills a significant void in Vietnamese NLP by providing a much-needed benchmark but also integrates seamlessly with the broader MTEB framework. The datasets are made publicly available on HuggingFace, fostering further research and development in the field of Vietnamese natural language processing. For a deeper dive into the methodology and results, the full research paper can be accessed at arXiv.
