TLDR: SmallThinker is a new family of large language models specifically designed for efficient local deployment on consumer devices like PCs and smartphones. Unlike traditional models adapted for the cloud, SmallThinker is built from the ground up with innovations like a two-level sparse architecture, pre-attention routing for faster data access, and memory-efficient attention mechanisms. This allows it to achieve state-of-the-art performance, including over 20 tokens per second on ordinary CPUs with minimal memory, often outperforming larger models without requiring expensive GPUs.
Large language models (LLMs) have become incredibly powerful, but their reliance on massive, GPU-powered cloud data centers has limited their reach. Imagine a future where these intelligent AI systems run directly on your everyday devices—your smartphone, your laptop, or even a small embedded device—without needing a constant internet connection or expensive specialized hardware. This is the vision behind SmallThinker, a new family of LLMs designed from the ground up for local deployment.
Unlike many existing approaches that try to shrink or adapt cloud-based models, SmallThinker embraces the limitations of local devices—such as weaker computational power, limited memory, and slower storage—as core design principles. This native approach allows SmallThinker to thrive in environments where traditional LLMs struggle.
Innovative Architecture for Local Performance
The innovation in SmallThinker lies in its deployment-aware architecture, which transforms these constraints into advantages. First, it introduces a unique two-level sparse structure. This combines a fine-grained Mixture-of-Experts (MoE) system with sparse feed-forward networks. In simple terms, instead of activating the entire model for every task, only the most relevant parts are used, drastically reducing the computational power needed without sacrificing the model’s overall knowledge or capability.
Second, to tackle the common issue of slow data access from storage (like an SSD), SmallThinker features a clever ‘pre-attention router’. This component predicts which parts of the model (called ‘expert parameters’) will be needed next, allowing them to be fetched from storage while the model is still busy with other calculations. This ‘prefetching’ effectively hides the storage delay, which would otherwise slow down on-device AI significantly.
Third, for memory efficiency, especially crucial for devices with limited RAM, SmallThinker uses a ‘NoPE-RoPE hybrid sparse attention’ mechanism. This smart design dramatically cuts down on the memory required for the ‘KV cache’ (which stores past computations to help the model understand long conversations), while still maintaining the ability to process long texts effectively.
Impressive Performance on Consumer Hardware
The results of these architectural innovations are quite remarkable. SmallThinker-4B-A0.6B and SmallThinker-21B-A3B, two models in the family, achieve state-of-the-art performance scores and can even outperform larger LLMs that require more resources. What’s truly impressive is their ability to run efficiently without high-end GPUs. With a common quantization technique (Q4_0), both models can process over 20 tokens per second on ordinary consumer CPUs, using only 1GB and 8GB of memory respectively. This means powerful AI can run on your existing computer or even a smartphone.
The paper highlights that SmallThinker-21B-A3B can be up to 86 times faster than Qwen3-30B-A3B, and SmallThinker-4B-A0.6B up to 19 times faster than Qwen3-1.7B, matching the speed of in-memory baselines. This efficiency extends to various devices, including PCs, smartphones (like the OnePlus 13), and even single-board computers like the Raspberry Pi 5, demonstrating consistent performance improvements.
Under the Hood: How It Works
The SmallThinker models are built on a standard Transformer architecture but with several key modifications. The fine-grained Mixture of Experts (MoE) architecture, with 32 experts for the 4B model and 64 for the 21B model, allows for highly efficient parameter use. Each expert uses a ‘ReGLU’ activation function, which further enhances sparsity, meaning even fewer computations are needed within each activated expert.
The ‘pre-attention router’ is strategically placed before the attention block, giving the system a head start in determining which experts are needed. This allows for parallel prefetching of expert weights while attention computations are ongoing. To ensure experts specialize in different tasks, a ‘DP-Groups Global Load Balance Loss’ is used during training. This promotes a hierarchical caching strategy where frequently used experts are kept in fast memory, while others reside on slower storage, optimizing access patterns.
The ‘NoPE-RoPE Hybrid Sparse Attention’ mechanism balances performance and memory by alternating between global attention (NoPE) and sliding window attention (RoPE) layers. This reduces the KV cache footprint without losing the ability to understand long contexts.
Training and Post-Training
SmallThinker models are pre-trained on a vast, high-quality dataset, primarily in English, including open-source web data, mathematical datasets, and coding corpora. They also use synthetically generated data, especially for math and code, to augment scarce high-quality sources. The training process involves a three-stage curriculum, gradually shifting from broad general data to more specialized and high-quality content, including instruction-response (SFT-style) data in the final stage.
After pre-training, the models undergo Supervised Fine-Tuning (SFT) on knowledge-intensive, code, and math data. A unique ‘model merging’ technique is also employed, linearly combining weights from different training stages to balance general knowledge with instruction-following accuracy.
The Inference Framework
To fully realize the on-device potential, SmallThinker integrates with the PowerInfer framework. This framework implements several optimizations: ‘Expert Offloading’ moves less frequently used expert parameters to SSDs, and an ‘Expert Prefetching Pipeline’ overlaps these I/O operations with computations. ‘Sparse Inference’ further enhances efficiency by selectively computing only the active neurons within ReGLU layers and identifying high-probability vocabulary rows in the language model head, eliminating unnecessary calculations.
This holistic system design transforms theoretical sparsity into real-world speedups, making SmallThinker a significant step towards truly capable AI on billions of devices worldwide. For more technical details, you can refer to the full research paper: SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment.
Also Read:
- DeltaLLM: Making Large Language Models Efficient for Edge Devices
- Optimizing Large Multimodal Models for Edge Devices with Adaptive Compression
Future Directions
While SmallThinker represents a major leap, the authors acknowledge areas for future improvement. The current pre-training dataset is smaller than those used for some state-of-the-art models, suggesting potential for broader knowledge and performance gains with larger datasets. Additionally, the models have only undergone supervised fine-tuning and could benefit from further alignment using Reinforcement Learning from Human Feedback (RLHF) to enhance response quality, helpfulness, and safety.
