TLDR: Researchers developed BioNAS, a new framework that uses Neural Architecture Search to automatically discover optimal combinations of different bio-inspired learning rules for each layer of a neural network. This approach significantly improves the accuracy of bio-inspired models on datasets like CIFAR-10 and ImageNet, making them competitive with traditional back-propagation methods, while also enhancing their inherent robustness against adversarial attacks.
Bio-inspired neural networks, which draw inspiration from the human brain’s structure and function, have garnered significant attention for their inherent advantages. These networks are known for their strong resistance to adversarial attacks, their energy efficiency, and their closer resemblance to how our brains work. However, they have historically faced challenges in matching the accuracy and scalability of traditional neural networks that rely on a method called back-propagation (BP).
A new research paper introduces a novel approach that aims to bridge this performance gap. The core idea is to allow different bio-inspired learning rules to be used in different layers of a neural network, rather than applying a single rule uniformly across the entire network. This layer-wise diversity is automatically discovered through a specialized procedure known as Neural Architecture Search (NAS).
Understanding the Core Concepts
At its heart, a neural network learns by updating its internal connections, or ‘weights.’ The method by which these weights are updated is called a ‘learning rule.’ Back-propagation is the most common learning rule in deep learning, but it has a biological limitation: it requires symmetric weight matrices for forward and backward passes, which is not observed in biological brains.
Bio-inspired learning rules, on the other hand, aim to be more biologically plausible. Examples include Feedback Alignment (FA), which uses random error signals; Direct Feedback Alignment (DFA), which sends feedback directly to each hidden layer; and various ‘sign-concordant’ feedback methods like Uniform Sign Feedback (uSF), Batchwise Random-Magnitude Sign Feedback (brSF), and Fixed Random Magnitude Sign-concordant Feedbacks (frSF). Other approaches like Hebbian learning, inspired by how neurons that fire together ‘wire together,’ and Predictive Coding, which iteratively reduces prediction errors, also fall into this category.
Neural Architecture Search (NAS) is an automated technique for designing neural networks. Instead of manually crafting an architecture, NAS explores a vast space of possibilities to find the best network structure for a given task. This paper builds upon existing NAS frameworks like DARTS (Differentiable Architecture Search) and EG-NAS (Evolutionary Architecture Search).
Introducing BioNAS: A New Framework
The researchers propose BioNAS, a framework that expands the traditional NAS search space to include not only different architectural components (like types of convolutional layers or pooling) but also various bio-inspired learning rules for each part of the network. This means BioNAS can automatically determine the optimal combination of network structure and the specific learning rule to apply at each layer.
The motivation behind this mixed-rule approach stems from recent neuroscience research suggesting that the brain itself might employ different learning mechanisms across different regions or layers. By mimicking this biological diversity, BioNAS aims to unlock better performance.
Impressive Results Across Benchmarks
The BioNAS framework has achieved remarkable results across several standard image classification datasets:
- On CIFAR-10, BioNAS-DARTS achieved a test error of 4.84%, significantly outperforming other bio-inspired methods and even surpassing some standard back-propagation baselines like ResNet20 and ResNet56.
- For CIFAR-100, it reached 23.52% test error.
- On ImageNet16-120, an accuracy of 43.42% was achieved.
- Most notably, on the challenging full ImageNet dataset, BioNAS-DARTS achieved a 60.51% top-1 accuracy. While still trailing the absolute best back-propagation models, this is a substantial improvement over previous bio-inspired models, which often struggled to break 30%.
These results highlight that jointly searching for both the network architecture and the layer-specific learning rules is crucial for maximizing performance.
Enhanced Adversarial Robustness
Beyond accuracy, a significant advantage of bio-inspired networks is their inherent robustness to adversarial attacks—subtle perturbations designed to fool AI models. The research shows that BioNAS-trained models exhibit superior resilience:
- Under ‘One-Pixel’ attacks (a black-box attack where only a few pixels are changed), BioNAS-DARTS consistently maintained high accuracy, whereas other models saw significant performance drops.
- For gradient-based attacks like FGSM, PGD, and APGD, BioNAS-DARTS maintained robust performance, while many models trained with single rules or standard back-propagation collapsed to near 0% accuracy under stronger attacks.
This enhanced robustness is attributed to the mixed learning rules disrupting the consistent gradient signals that attackers typically exploit, making it harder to craft effective adversarial examples.
Insights into Learning Dynamics
The study also delved into why mixing learning rules is so effective. It was observed that BioNAS-DARTS models trained with mixed rules exhibited consistently lower ‘gradient variance’ compared to models trained with a single fixed rule. Lower gradient fluctuations can lead to more stable and efficient optimization during training, helping the network navigate the complex ‘loss landscape’ more effectively.
Furthermore, the benefits of mixing rules don’t seem to depend on a specific pattern or combination of rules; even random assignments of different rules across layers maintained high accuracy, suggesting that the diversity itself is key.
Also Read:
- Bridging the Forgetting Divide in AI: A Brain-Inspired Approach to Continual Learning
- Spiking Neural Networks: Bringing Brain-Inspired AI to the Edge
A Promising Path Forward
This research demonstrates that by intelligently combining neural architecture search with diverse bio-inspired learning rules, it’s possible to create neural networks that achieve high accuracy comparable to back-propagation models, while retaining and even enhancing the natural robustness advantages of bio-inspired systems. This framework offers a promising direction for developing more biologically plausible and robust deep learning models, particularly for researchers interested in modeling brain functions. For more details, you can read the full paper here.
