TLDR: MDK12-Bench is a new large-scale benchmark for evaluating Multimodal Large Language Models (MLLMs) using real K-12 school exams across six subjects. It features 141,000 questions, detailed knowledge points, and innovative dynamic evaluation methods to test model generalization and prevent data contamination. Findings show current MLLMs struggle with harder, newer, and dynamically altered questions, especially in math and physics, and that simply adding knowledge points doesn’t significantly help with complex reasoning tasks.
Multimodal Large Language Models, or MLLMs, are a significant step towards achieving Artificial General Intelligence (AGI) by combining language and visual understanding to solve problems. However, evaluating these advanced AI models has been challenging due to limitations in existing benchmarks. Many current evaluation tools are too small, cover only a narrow range of topics, or use static tests that can become outdated as models are trained on more data.
To address these issues, researchers have introduced MDK12-Bench, a new, extensive benchmark designed to thoroughly evaluate MLLMs. This benchmark is built from real-world K-12 (kindergarten to 12th grade) exams, covering six different subjects: Mathematics, Physics, Chemistry, Biology, Geography, and Information Science. It boasts an impressive 141,000 unique questions and is linked to 6,225 specific knowledge points, organized in a detailed six-layer structure.
MDK12-Bench offers a comprehensive evaluation by considering several dimensions: how models perform across different difficulty levels, their adaptability to changes over time (cross-year shifts), their ability to handle new contexts, and their skill in knowledge-driven reasoning. The benchmark includes five different question formats and provides annotations for difficulty and the year of the exam, allowing for a more nuanced assessment of MLLM capabilities.
A key innovation of MDK12-Bench is its dynamic evaluation framework. This framework introduces unfamiliar visual, textual, and question format changes during testing. This approach helps to rigorously test a model’s ability to generalize to new situations and reduces the risk of data contamination, where models might perform well simply because they’ve already seen similar data during training. The researchers also explored Knowledge-Point Reference-Augmented Generation (KP-RAG), which involves providing models with relevant knowledge points to see how this additional information aids in problem-solving.
Experiments conducted on various state-of-the-art MLLMs, including both proprietary and open-source models, revealed several important findings. Larger models generally performed better across subjects and difficulty levels, showing improved visual understanding. However, this increase in size didn’t always translate to significant gains in reasoning accuracy. Models specifically optimized for reasoning tasks showed better reasoning accuracy than general chat models, but not necessarily better visual perception.
The study found that MLLMs struggled more with Mathematics and Physics questions compared to other subjects. Performance also declined significantly on harder and newer exams, suggesting that models have difficulty generalizing to more complex or recently introduced concepts. The dynamic evaluation framework caused a notable drop in performance, highlighting the models’ limitations in generalizing to unexpected contextual changes. Interestingly, while KP-RAG improved accuracy on easier exams (where factual recall is more important), its benefits were limited on harder tasks that require multi-step reasoning rather than just factual knowledge.
Also Read:
- Assessing AI’s Geometry Skills: A New Benchmark for Complex Problems
- OmniBench-RAG: A New Standard for Evaluating Retrieval-Augmented Generation
In conclusion, MDK12-Bench serves as a vital tool for understanding the current strengths and weaknesses of MLLMs. Its large scale, multidisciplinary coverage, and innovative dynamic evaluation methods provide a robust foundation for diagnosing model limitations and guiding the development of more adaptable, robust, and generalizable multimodal AI. For more technical details, you can refer to the full research paper here.
