TLDR: QuarterMap is a novel post-training activation pruning method designed to enhance the efficiency of Visual State Space Models (SSMs) like VMamba. It achieves significant throughput improvements (up to 11% speedup) with minimal accuracy loss (less than 0.9%) by removing redundant spatial activations before scanning and restoring dimensions via nearest-neighbor upsampling. This technique is particularly effective for VMamba’s four-directional scanning mechanism and requires no retraining, making it a practical solution for deployment-time optimization in computer vision and medical imaging.
In the rapidly evolving world of artificial intelligence, especially in computer vision, models are becoming increasingly powerful but also more computationally demanding. Transformers, a popular architecture, are known for their ability to understand complex relationships in data, but they come with a significant cost: their computational complexity grows quadratically with the size of the input. This means as images get larger or more detailed, the processing time can skyrocket.
To address this, researchers have been exploring alternatives, and State Space Models (SSMs) have emerged as a promising solution. SSMs offer a more efficient way to process data by using linear recurrence, which is much faster than the quadratic complexity of transformers. Among these, VMamba has recently gained attention as a strong vision backbone, showing impressive performance in tasks like image classification.
However, even VMamba, despite its efficiency, faces a bottleneck. Its unique four-directional scanning mechanism, while powerful for capturing spatial information, introduces a lot of redundant data. This redundancy slows down the model, making it less ideal for real-world deployment where speed is crucial.
Introducing QuarterMap: A Smart Way to Speed Up VMamba
A new method called QuarterMap has been proposed to tackle this very challenge. QuarterMap is a post-training activation pruning technique, which means it optimizes the model *after* it has already been trained, without needing to retrain it from scratch. This is a significant advantage, as retraining large AI models can be incredibly time-consuming and resource-intensive.
The core idea behind QuarterMap is to remove redundant spatial activations (pieces of data) before VMamba’s scanning process begins. Once the essential information has been processed, the dimensions are restored using a simple nearest-neighbor upsampling technique. This clever approach allows QuarterMap to significantly improve the model’s throughput (how many images it can process per second) without causing a noticeable drop in accuracy.
How QuarterMap Works Its Magic
QuarterMap operates through a lightweight two-stage process. First, it applies spatial pruning before the VMamba’s cross-scan module. During this pruning stage, QuarterMap selectively keeps only every alternate element in both the horizontal and vertical dimensions of the image data. This works well because VMamba’s four-way scanning naturally creates a lot of overlapping and redundant information, which QuarterMap efficiently identifies and removes.
After the pruned data goes through VMamba’s processing, QuarterMap uses nearest-neighbor upsampling to bring the spatial dimensions back to their original size. This method is chosen because it’s computationally inexpensive and effective, based on the assumption that neighboring pixels in an image often contain similar information. By reducing the data size before the most computationally intensive parts of VMamba, QuarterMap achieves substantial speed improvements.
Impressive Results Across Various Tasks
The effectiveness of QuarterMap has been demonstrated across different computer vision tasks. On the ImageNet-1K benchmark, QuarterMap achieved up to an 11% speedup on VMamba with less than a 0.9% drop in accuracy. This is a remarkable trade-off, especially when compared to other token merging methods like ToMe, which often incur higher overheads due to complex merge-unmerge operations.
Beyond general image classification, QuarterMap also showed similar gains in semantic segmentation tasks (where the model identifies and outlines objects in an image). Furthermore, it proved highly effective on MedMamba, a specialized model built on VMamba’s architecture for medical imaging. In medical imaging tasks like classifying blood, organ, retina, and pathology images, QuarterMap consistently improved throughput by about 21% without any degradation in classification accuracy. This highlights its versatility and applicability in critical domain-specific deployments.
It’s important to note that QuarterMap is particularly well-suited for VMamba and models that share its four-directional scanning structure. When tested on other architectures like Convolutional Neural Networks (CNNs) or Vision Transformers (ViTs), QuarterMap did not perform as well, indicating its specialized design for the unique redundancy patterns found in VMamba.
Also Read:
A Plug-and-Play Solution for Efficient AI Deployment
In conclusion, QuarterMap offers a valuable tool for making Visual State Space Models more efficient at deployment time. Its ability to improve runtime efficiency with minimal accuracy loss and without requiring any retraining makes it a ‘plug-and-play’ solution. This is especially beneficial in scenarios where computational resources are limited or when working with pre-trained models. QuarterMap not only advances the field of efficient AI but also opens new avenues for exploring pruning strategies in State Space Models. You can read the full research paper here.
