TLDR: A new training-free method called Multi-basis Exponential Decay (MBE) neuron is proposed for converting Artificial Neural Networks (ANNs) to Spiking Neural Networks (SNNs), specifically for Transformer architectures. This method effectively handles complex nonlinear operations, achieves near-lossless conversion accuracy with significantly lower latency across various tasks (CV, NLU, NLG) and models (ViT, RoBERTa, GPT-2), and offers substantial energy savings compared to traditional ANNs.
Artificial Neural Networks (ANNs) have achieved remarkable success in various domains, but their computational demands, especially for large models like Transformers, pose significant challenges for energy-constrained devices. Spiking Neural Networks (SNNs) offer a compelling alternative, leveraging an event-driven, sparse spike-based computing paradigm that promises superior energy efficiency.
While directly training SNNs can be complex and computationally expensive, converting pre-trained ANNs to SNNs (ANN-to-SNN conversion) bypasses these high training costs. However, existing conversion methods have struggled to effectively handle the intricate nonlinear operations prevalent in Transformer architectures, often requiring additional fine-tuning or leading to notable performance degradation.
A recent research paper, titled “Training-Free ANN-to-SNN Conversion for High-Performance Spiking Transformer,” introduces a novel framework that addresses these limitations. Authored by Jingya Wang, Xin Deng, Wenjie Wei, Dehao Zhang, Shuai Wang, Qian Sun, Jieyuan Zhang, Hanwen Liu, Ning Xie, and Malu Zhang, the paper proposes a high-performance and training-free ANN-to-SNN conversion framework specifically designed for Transformer architectures. You can read the full paper here.
The core innovation lies in the introduction of a Multi-basis Exponential Decay (MBE) neuron. Unlike previous approaches, the MBE neuron employs an exponential decay strategy and a multi-basis encoding method. This allows it to efficiently approximate a wide range of nonlinear operations found in Transformers, such as GELU, Tanh, floating-point multiplications, Softmax, and LayerNorm, without needing to modify the weights of the pre-trained ANNs. This is a significant advancement, as it removes the requirement for additional fine-tuning processes that often add computational overhead.
The researchers systematically analyzed the shortcomings of prior methods, particularly the “Few-spikes (FS) neuron,” identifying issues like excessive dependence on initialization and global suboptimality in approximating complex functions. The MBE neuron was designed to overcome these challenges, offering improved expressiveness and stability.
The proposed conversion framework based on MBE neurons enables a fully spike-driven representation of Transformer components. For instance, floating-point multiplications are transformed into spike-based Hadamard products, and complex operations like Softmax and LayerNorm are decomposed into basic functions (exponential, reciprocal, square root) that are then approximated by specialized MBE neurons. This ensures that the converted SNNs maintain their crucial spike-based property, avoiding the high energy consumption associated with traditional floating-point operations during inference.
Extensive experiments were conducted across diverse tasks, including computer vision (CV), natural language understanding (NLU), and natural language generation (NLG), and on mainstream Transformer architectures like ViT, RoBERTa, and GPT-2. The results demonstrate that this new method achieves near-lossless conversion accuracy, meaning the SNNs perform almost as well as their original ANN counterparts, but with significantly lower latency and improved energy efficiency. For example, on ImageNet, ViT models showed conversion losses of less than 1%, and on NLU tasks, RoBERTa models achieved competitive accuracy with substantially fewer timesteps compared to other methods.
The energy estimation further highlights the framework’s benefits, showing that the converted SNNs can achieve an energy ratio as low as 17.29% compared to the original ANNs for models like ViT-M/16. This is attributed to the sparse spike-driven nature of SNNs and the efficient encoding of information by the MBE neurons.
Also Read:
- Fourier-VLM: A New Approach to Efficient Vision-Language Models
- Enhanced LLM Performance on AI PCs with New Inference Runtime
In conclusion, this training-free ANN-to-SNN conversion framework for Transformer architectures represents a promising pathway for the efficient and scalable deployment of high-performance Spiking Transformers in real-world applications, bridging the gap between powerful deep learning models and energy-efficient neuromorphic hardware.
