TLDR: The paper “Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning” introduces a novel framework that significantly improves mathematical reasoning in multilingual large language models (LLMs). By training a reward model to rank generated responses across various languages, the research demonstrates that leveraging the complementary reasoning skills of diverse languages leads to enhanced performance. This approach benefits even high-resource languages like English, especially under low sampling budgets, by exploiting the unique problem-solving perspectives offered by different linguistic contexts.
Large language models (LLMs) are becoming increasingly adept at complex reasoning tasks, but how their abilities vary across different languages, especially in multilingual models, has been a less explored area. A new research paper, “Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning,” delves into this question, proposing an innovative approach to harness the diverse strengths of multiple languages for improved mathematical problem-solving.
Traditionally, much of the research on improving LLM reasoning has focused on English. While multilingual LLMs have shown promise, the idea of combining reasoning paths from different languages to achieve a better outcome was largely unexplored. This paper highlights a fascinating insight: even languages typically considered ‘low-resource’ can sometimes succeed where ‘high-resource’ languages, including English, might fail. This suggests that different languages offer unique, complementary reasoning signals that can be incredibly valuable.
The core of this research is a novel framework called cross-lingual outcome reward modeling (ORM). In simple terms, a ‘reward model’ acts like a judge, evaluating the correctness of an AI’s generated answer. While some reward models assess reasoning step-by-step (process-based), this work focuses on outcome reward models, which evaluate the final answer and its entire reasoning path. The ‘cross-lingual’ aspect means this judge doesn’t just evaluate answers within a single language; it ranks responses generated across multiple languages for the same question and then selects the best one.
To train this cross-lingual verifier, the researchers used the well-known GSM8K dataset of grade school math problems. This dataset was translated into eight languages: English, Spanish, French, German, Russian, Chinese, Japanese, and Thai. Responses were generated by several different LLMs (Aya-Expanse 8B, Llama3.1 8B, and Qwen2.5 7B) for these translated problems. These generated reasoning paths were then automatically labeled as correct or incorrect, creating a diverse training set of around 88,000 samples. The multilingual Qwen2.5-Instruct 3B model was chosen as the reward model due to its extensive language support and was fine-tuned using this dataset.
The experiments were conducted using the MGSM (Multilingual Grade School Math) dataset, covering 11 languages, with 250 examples for each. The models tested included a range of instruction-tuned multilingual LLMs like Aya-Expanse 8B, Llama3.1 8B, Qwen2.5 7B, Ministral 8B, Phi3 7B, and Llama 3.2 3b. The cross-lingual ORM framework was compared against baselines such as self-consistency (where the model takes a majority vote from multiple answers in one language) and multilingual ORM (where a verifier scores multiple samples within a single language).
The findings were compelling. The research demonstrated that LLMs indeed possess complementary mathematical reasoning skills across languages. A key metric, pass@8-Cross, which considers a question solved if at least one answer across different languages is correct, significantly outperformed pass@8-Multi, which averages scores within individual languages. This strongly suggests that even high-resource languages can benefit from insights gleaned from other languages.
Furthermore, the cross-lingual ORM framework consistently outperformed the average performance of multilingual ORM, with non-English languages showing the most significant improvements. This indicates that leveraging signals across languages is more effective than relying solely on monolingual reasoning. The study also found that increasing the number of languages in the pool generally enhanced the cross-lingual ORM’s performance, up to a certain point.
Interestingly, cross-lingual sampling also benefited English, particularly when the sampling budget was low. While English often shows the highest performance in multilingual models, combining it with other languages proved advantageous in scenarios where fewer samples were available. The researchers also observed that while including English in the language pool generally helps, some combinations of non-English languages could even outperform certain groups that included English, further reinforcing the idea of diverse, complementary reasoning skills.
Also Read:
- Unlocking Efficiency in Language Models: A New Bias-Selection Method for Fine-Tuning
- Foundation Models Navigate Virtual Worlds: New Strategies for Reinforcement Learning
In conclusion, this paper introduces a powerful cross-lingual reward modeling framework that effectively leverages the complementary mathematical reasoning skills present across different languages in multilingual LLMs. The results open new avenues for improving multilingual reasoning by recognizing and utilizing the unique strengths each language brings to problem-solving. While the current work focused on math reasoning and a specific set of languages, it paves the way for future research into broader applications and deeper understanding of cross-lingual reasoning patterns. You can read the full paper here: Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning.
