TLDR: A novel method called ‘reasoning vectors’ is introduced to enhance the reasoning capabilities of Japanese Large Language Models (LLMs) without extensive additional training or data. By extracting the difference in weights between a pre-trained and a reasoning-tuned English LLM, this ‘reasoning vector’ is then added to a Japanese LLM, significantly improving its performance on benchmark tasks like mathematical problem-solving. This approach offers a resource-efficient way to boost under-resourced language models.
Large Language Models (LLMs) have made incredible strides, especially with advanced post-training techniques like supervised fine-tuning and reinforcement learning. These methods significantly boost performance and reasoning abilities. However, applying these same powerful techniques to Japanese LLMs presents unique challenges. The main hurdles include a scarcity of public datasets, a limited number of expert annotators, and the absence of robust, large-scale Japanese models to effectively filter and assess data quality. Relying on machine-translated data from English can also be problematic, as translations often miss the subtle linguistic nuances and cultural context vital for native Japanese understanding.
A Novel Approach: Reasoning Vectors
To overcome these obstacles, researchers have explored a novel method: extracting ‘reasoning vectors’ from mainstream LLMs and transferring them to Japanese models. This approach aims to enhance the reasoning capabilities of Japanese LLMs without requiring extensive additional training or new datasets. The concept is inspired by ‘task vectors,’ which capture the change in model weights before and after training for a specific task.
The core idea is straightforward: a reasoning vector is obtained by calculating the difference in weights between a pre-trained model and a post-trained model that has been fine-tuned for reasoning. This vector essentially represents the ‘direction’ of reasoning knowledge in the model’s weight space. Once obtained, this reasoning vector is then added to a target Japanese LLM, effectively injecting the acquired reasoning capabilities into it.
How the Experiment Was Conducted
For their experiments, the researchers used Qwen-32B as the pre-trained model and s1-32B as the post-trained reasoning model. EZO, a Japanese instruction-tuned model, served as the target model. These specific models were chosen because they shared the same underlying architecture, which is crucial for the element-wise arithmetic operations involved in creating and applying the reasoning vector.
To evaluate the effectiveness of this method, the models were tested on the American Invitational Mathematics Examination (AIME24) dataset. This dataset, known for its challenging problems, was manually translated into Japanese. For grading, a simple heuristic was used: a response was considered correct if it contained the answer label. While simple, this method provided a clear indication of performance changes.
Promising Results
The results were highly encouraging. The study found that incorporating the reasoning vector into the target Japanese model consistently led to performance gains. Even with a small scalar weight (w = 0.25) applied to the vector, the enhanced Japanese model showed a slight improvement over both the original pre-trained and target models. As the weight increased, the performance improved further. Notably, at higher weights, the enhanced Japanese model even surpassed the original post-trained English reasoning model in terms of correct answers.
These findings underscore the effectiveness of reasoning vector integration in significantly improving task performance. It suggests that the enhanced Japanese model successfully acquired reasoning capabilities from the transferred vector, demonstrating a simple yet powerful way to boost under-resourced language models by leveraging advancements in other languages.
Also Read:
- Enhancing LLM Reasoning with Consistency-Aware Policy Optimization
- SMEdit: Enhancing Large Language Model Editing with Multi-Step Learning
Looking Ahead
While this method offers a promising path, the researchers acknowledge certain limitations. The primary constraint is that the pre-trained, post-trained, and target models must share the same architecture for the vector operations to work. Additionally, for evaluation datasets without a dedicated training set, determining the optimal weight for the reasoning vector can be challenging. Future work will explore applying this approach to smaller and quantized models to assess its broader feasibility. For more details, you can read the full research paper here.
