Step-3: A New Era of Cost-Effective Large Language Model Decoding

TLDR: Step-3 is a 321-billion parameter VLM from StepFun Inc. that significantly reduces LLM decoding costs through a novel model-system co-design. It introduces Multi-Matrix Factorization Attention (MFA) for efficient KV cache and computation, and Attention-FFN Disaggregation (AFD) to decouple attention and FFN layers for optimized hardware utilization. This approach leads to substantially lower theoretical decoding costs and higher throughput (up to 4,039 tokens/GPU/s) compared to competitors like DeepSeek-V3, demonstrating that hardware-aligned design, MoE sparsity, and AFD are key to cost-effectiveness, rather than just parameter count.

Large language models (LLMs) are powerful, but making them run efficiently, especially for complex tasks that require long periods of ‘thinking’ or ‘decoding’, is a major challenge. This process often consumes a lot of hardware resources, leading to high operational costs.

StepFun Inc. has introduced a new approach with their Step-3 model, a 321-billion parameter Vision-Language Model (VLM), specifically designed to minimize these decoding costs. Their innovation lies in a ‘model-system co-design’ strategy, which means they’ve optimized both the model’s architecture and the underlying inference system to work together seamlessly.

Key Innovations for Cost-Effective Decoding

Step-3 features two primary breakthroughs:

1. Multi-Matrix Factorization Attention (MFA): This is a new way the model handles ‘attention’ – how it focuses on different parts of the input. MFA significantly reduces the size of the ‘KV cache’ (memory used to store past information) and the amount of computation needed, all while keeping the model highly expressive and accurate. This is crucial because attention layers are often memory-intensive during inference.

2. Attention-FFN Disaggregation (AFD): This is a distributed inference system that separates the attention layers from the Feed-Forward Network (FFN) layers. Think of it as having specialized teams for different parts of the model’s work. Attention layers, which are memory-heavy, can be run on hardware best suited for memory, while FFN layers, which are computation-heavy, can run on hardware optimized for raw processing power. This decoupling allows each part to operate under ideal hardware conditions, maximizing efficiency.

Unprecedented Cost Efficiency and Performance

This co-design approach allows Step-3 to achieve remarkable cost reductions compared to other large models like DeepSeek-V3 and Qwen3 MoE 235B. The cost savings become even more significant when dealing with longer text contexts. Despite activating 38 billion parameters per token (more than DeepSeek-V3 and Qwen3 MoE 235B), Step-3 manages to keep costs low. This demonstrates that aligning the model’s design with hardware capabilities, smart use of Mixture-of-Experts (MoE) sparsity, and AFD are vital for cost-effectiveness.

In head-to-head comparisons, Step-3 shows impressive real-world performance. On Hopper GPUs, it achieves a decoding throughput of up to 4,039 tokens per second per GPU, significantly higher than DeepSeek-V3’s 2,324 tokens per second in the same setup. This sets a new standard for LLM decoding efficiency.

Why Decoding Optimization Matters

Optimizing decoding is critical for several reasons: it’s the most expensive part of running LLMs per token, especially for reasoning models where longer ‘thinking’ leads to better intelligence. Lowering these costs can directly translate to more intelligent models for the same budget. Faster and cheaper decoding also accelerates the training of reinforcement learning models.

Addressing Suboptimal Practices in Other Models

The researchers identified common issues in existing large open-weight models:

Some attention designs reduce KV cache size at the expense of excessive computation, making them less cost-effective on more affordable hardware.
Some FFN designs pursue sparsity without considering if they fit today’s hardware, harming efficiency or performance.

Step-3’s design aims to inspire a re-evaluation of these trends by offering a more balanced and hardware-aware approach.

The Power of Attention-FFN Disaggregation (AFD)

AFD is a core component of Step-3’s inference system. By deploying attention and FFN layers on separate sets of GPUs, it allows for different parallelism strategies tailored to each component’s needs. This separation also enables independent analysis and optimization of attention and FFN, leading to more flexible model architecture modifications.

Compared to existing solutions like DeepSeek-V3’s Expert Parallelism (EP), AFD offers several advantages: it operates efficiently at a smaller deployment scale, handles long contexts more effectively, reduces load imbalance issues, supports heterogeneous hardware (mixing different types of GPUs), and provides clearer performance modeling.

Beyond Parameter Count: True Cost Indicators

The paper highlights that neither the total parameter count nor the activated parameter count is a good indicator for decoding costs. For instance, Qwen-3 MoE 235B has significantly fewer parameters than DeepSeek-V3 and Step-3, yet its theoretical decoding cost is not proportionally lower. Step-3, despite having the highest activated parameters, achieves substantial cost reductions. This is because attention design, not just parameter count, dominates decoding costs, especially as context length grows.

Step-3’s MFA design is also more ‘hardware-friendly’ across various GPU types, showing minimal cost differences on weaker hardware compared to other models that see multi-fold cost increases.

Also Read:

The Role of StepMesh

To facilitate the complex communication required by AFD, StepFun developed StepMesh, a specialized communication library. StepMesh ensures ultra-low latency and efficient data transfer between attention and FFN instances, crucial for maintaining the high throughput and low latency targets of the system. It supports heterogeneous accelerators and is designed to co-evolve with network technologies.

For more technical details, you can refer to the full research paper here: Step-3 Research Paper.