TLDR: PLaMo 2 is a new series of Japanese-focused large language models by Preferred Networks. It features a hybrid Samba-based architecture that evolves to full attention for 32K token contexts, leverages extensive synthetic data to overcome language-specific data scarcity, and employs efficient training methods like weight reuse and pruning. The models achieve state-of-the-art performance on Japanese benchmarks, with an 8B model matching a previous 100B model, and are optimized for inference through vLLM and quantization.
Preferred Networks has unveiled PLaMo 2, a new series of large language models specifically designed to excel in the Japanese language. This technical report introduces a multifaceted approach that addresses key challenges in LLM development, including computational cost, efficient inference for long sequences, and the scarcity of high-quality training data for non-English languages.
A Hybrid Architecture for Enhanced Performance
PLaMo 2 adopts a novel hybrid architecture inspired by Samba, which combines the Mamba state space model with sliding window attention. This design prioritizes computational efficiency and scalability, aiming to overcome the limitations of traditional Transformer models, particularly concerning long sequence lengths and memory consumption during inference. Initially, the architecture balances efficient sequential processing with the ability to capture local dependencies. Crucially, it transitions to a full attention mechanism during a continual pre-training phase to support extensive 32K token contexts, addressing long-context retrieval limitations observed in earlier hybrid designs.
Overcoming Data Scarcity with Synthetic Data
A significant innovation in PLaMo 2 is its extensive use of high-quality synthetic data generated by other large language models. This strategy is vital for languages like Japanese, where large-scale, high-quality training datasets are less abundant compared to English. The synthetic data generation focuses on diverse tasks such as translation, paraphrasing, code generation, and mathematical problem-solving. For instance, high-quality English data is translated into Japanese, and existing Japanese data is paraphrased to augment the dataset. Code data is generated by adding comments and creating paraphrased versions, while mathematical problems and their solutions are created using existing datasets as seeds.
Efficient Training and Model Scaling
PLaMo 2 employs an efficient training paradigm that includes weight reusing and structured pruning. Weight reusing allows larger models to be initialized using weights from smaller models, leading to significant performance gains and faster training loss reduction. Pruning, on the other hand, enables the creation of high-performance, compact models from larger parent models by retaining only the most critical weights. This methodology is exemplified by the PLaMo 2.1-8B model, which was developed from the PLaMo 2-31B model and achieves performance comparable to or even exceeding the previous 100B-parameter PLaMo-100B model, showcasing remarkable efficiency in model scaling.
Post-Training Refinements and Long-Context Support
The post-training pipeline for PLaMo 2 involves supervised fine-tuning (SFT) and direct preference optimization (DPO), enhanced by synthetic Japanese instruction data and advanced model merging techniques. A dedicated continual pre-training (CPT) phase was implemented to specifically improve the model’s ability to handle long contexts, expanding the sliding window attention to full attention to support up to 32K tokens. This ensures the model can accurately retrieve and extract specific information from lengthy inputs, a critical capability for complex tasks.
Optimized Inference for Practical Deployment
To ensure practical deployment and reduce computational costs, PLaMo 2 models are optimized for inference. This includes leveraging the vLLM framework for efficient memory management and customizing it to support the hybrid architecture. Further optimizations involve chunked prefill for balanced workload distribution and selective integration of torch.compile for enhanced execution efficiency. Additionally, quantization techniques, such as INT4 for model weights and FP8 for the KV cache, significantly reduce memory footprint and computational requirements, making it possible to run large models on more accessible hardware like single L40S GPUs with minimal accuracy loss.
Also Read:
- SpikingBrain: Brain-Inspired Models for Efficient Large Language Processing
- Unlocking Efficiency in Large Language Model Pretraining: Practical Scaling Strategies
State-of-the-Art Performance on Japanese Benchmarks
Comprehensive evaluations demonstrate that the PLaMo 2 models, particularly PLaMo 2.1-8B and PLaMo 2.0-31B, achieve state-of-the-art results across a wide array of Japanese benchmarks. They consistently outperform similarly-sized open models in instruction-following, language fluency, and Japanese-specific knowledge, as measured by benchmarks like Jaster, M-IFEval Japanese, and pfgen-bench. While excelling in many areas, the report acknowledges that there is room for future improvement in complex multi-step and mathematical reasoning tasks. For more in-depth technical details, you can refer to the full PLaMo 2 Technical Report.
