TLDR: BlueLM-2.5-3B is a compact, 2.9B-parameter multimodal AI model from vivo AI Lab, designed for efficient edge device deployment. It uniquely supports “thinking” and “non-thinking” modes with controllable token budgets. Developed with advanced data curation, hybrid reinforcement learning, and optimized infrastructure, it achieves strong multimodal and text performance comparable to larger models, demonstrating high data efficiency.
The vivo AI Lab has unveiled BlueLM-2.5-3B, a new compact and unified dense Multimodal Large Language Model (MLLM) specifically designed for efficient deployment on edge devices like mobile phones, vehicles, and robots. This model stands out as the first 3-billion-parameter scale MLLM to support both “thinking” and “non-thinking” modes, offering explicit control over the thinking token budget, which is crucial for managing latency on resource-constrained devices.
BlueLM-2.5-3B was developed using a sophisticated approach that includes diversified data curation, strategic data resampling, a hybrid heterogeneous reinforcement learning framework, and a high-performance training infrastructure. Despite its compact size of only 2.9 billion parameters, the model achieves impressive multimodal capabilities while maintaining strong performance in pure-text tasks.
In its “thinking mode,” BlueLM-2.5-3B demonstrates performance comparable to larger models like Qwen3-4B on text-only benchmarks. For multimodal evaluations, it trails the much larger Kimi-VL-A3B-16B by only about 5% on average. When operating in “non-thinking mode,” it surpasses Qwen2.5-VL-3B on most multimodal benchmarks. A notable achievement is its exceptional data efficiency, as it achieves this performance with significantly less total training data compared to Qwen2.5-VL-3B and Qwen3-4B.
Model Architecture and Training Innovations
The architecture of BlueLM-2.5-3B consists of three main components: a Vision Transformer (ViT) for processing visual data, an adapter module to align visual and language representations, and a dense Large Language Model (LLM) backbone. The model is designed for compactness, featuring 22% fewer parameters than comparable models like Qwen2.5-VL-3B.
The training process is multi-phased, starting with pure-text pre-training to initialize the LLM, followed by a multimodal pre-training phase. This includes a joint general pre-training stage, a reasoning-enhanced stage that incorporates a substantial amount of synthesized image-text reasoning data, and a joint fast-decay and long-context activation stage to improve long-context capabilities. Post-training stages involve supervised fine-tuning with a unique “thinking mode fusion” mechanism, allowing the model to adapt its reasoning based on user queries. A special token, [ |BlueThink|], acts as a control switch for activating the long-thinking mode, which has shown remarkable stability with a failure rate of less than 1 PPM.
Reinforcement Learning from Human Feedback (RLHF) and Reinforcement Learning with Verifiable Rewards were also integrated to refine the model’s responses, particularly for open-ended questions and reasoning problems. An innovative length penalty mechanism, termed “Group Overlong,” helps reduce redundant reasoning steps, making the model more efficient for edge deployment by curbing “overthinking” behavior.
Also Read:
- Unlocking Smarter AI: How Large Language Models Are Learning to Reason on a Budget
- Unlocking Deeper Reasoning: How ‘Modular Thinking’ is Revolutionizing LLMs
Data and Infrastructure
The model’s robust performance is underpinned by a meticulously constructed data pipeline. The pre-training data has been significantly expanded, with pure text accounting for 40% of the multimodal pre-training data. This includes diverse categories such as image captioning, OCR, Visual Question Answering (VQA), and Graphical User Interface (GUI) data. A large-scale corpus specifically designed for reasoning tasks, including synthetic data, further enhances the model’s capabilities.
The training infrastructure boasts an on-premises GPU cluster with thousands of high-performance GPUs, interconnected by a high-efficiency InfiniBand network. An in-house training framework based on Megatron-LM was developed, focusing on efficiency, scalability, stability, and observability. Innovations like multi-sample concatenation and context parallelism for long-sequence training significantly boost throughput. The RL infrastructure also saw customized development and performance optimization, including one-step asynchronous RL and inference engine load balancing.
The comprehensive evaluation across over 20 benchmark datasets confirms BlueLM-2.5-3B’s strong performance in both multimodal and pure-text reasoning, visual perception, and GUI agent grounding. This work by vivo AI Lab represents a significant step towards high-performance, on-device MLLMs. For more technical details, you can refer to the full research paper: BlueLM-2.5-3B Technical Report.
