TLDR: A new benchmark, MateInfoUB, tests LLMs on competitive, multilingual, and multimodal computer science problems. It finds newer LLMs often outperform students, highlights language and reasoning strategy impacts, and raises ethical concerns for education, while making the dataset publicly available.
Large Language Models (LLMs) are rapidly changing various fields, including computer science education. While these models show impressive abilities in coding and problem-solving, their full potential and limitations in advanced computer science contexts are still being explored.
A new study introduces MateInfoUB, a unique real-world benchmark designed to thoroughly test LLMs. This benchmark features a bilingual (English-Romanian) and multimodal (text and image) dataset of multiple-choice questions. These questions are derived from a high-level computer science competition, making them particularly challenging. A key aspect of this dataset is that some problems are better solved with traditional reasoning on paper, while others are more efficiently tackled by writing code. This setup forces LLMs to decide the best approach autonomously.
The MateInfoUB dataset originates from an annual computer science contest for 12th-grade students, which also serves as an admission exam for the University of Bucharest’s Faculty of Mathematics and Computer Science. The researchers focused on the first phase of this contest, which involves challenging multiple-choice questions. The dataset includes 100 questions, complete with detailed solutions in Romanian, and has been translated into English with manual verification.
The study systematically evaluated several state-of-the-art LLMs, including models from Google AI Studio, Mistral AI, and Together AI. The findings reveal significant insights into LLM performance:
Also Read:
- Rethinking LLM Evaluation: Why ‘Answer Matching’ Outperforms Multiple Choice
- Unlocking Advanced Math Skills in LLMs: The Power of Diversified Thinking
Key Findings from the MateInfoUB Benchmark:
- LLMs vs. Human Performance: Newer LLMs consistently outperformed most high school students who participated in the actual contest, indicating their growing proficiency in complex computer science tasks.
- Language Impact: Most models performed better on the original Romanian versions of the questions compared to their English translations. This suggests that translation quality and prior exposure to the original language can influence performance.
- Multiple-Choice Assistance: The presence of multiple-choice options slightly improved model accuracy, likely by providing some guidance or increasing the chance of a correct guess.
- Reasoning Strategies: A “hybrid” approach, where LLMs could autonomously choose between generating code or providing direct reasoning, showed promising results, often outperforming methods that relied solely on one strategy.
The research also highlights critical ethical considerations regarding educational integrity and fair assessments in the age of LLMs. As LLM capabilities advance, educators and contest organizers need to adapt assessment methods to prevent unauthorized use and ensure fairness. The study suggests combining tasks that require tool use with those demanding manual reasoning to accurately gauge student understanding.
To support further research and practical application, the MateInfoUB dataset is being made publicly available in both English and Romanian. Additionally, an educational application tailored for Romanian students has been released, allowing them to practice and self-assess using the dataset in an interactive environment. You can explore the full details of this research paper here.
