TLDR: G2GNet is a new artificial neural network architecture inspired by how groups of neurons communicate in the mouse visual cortex. It uses sparse, modular connections and dynamically prunes and regrows connections based on a “Hebbian-inspired” rule. This approach significantly reduces the number of parameters (up to 75% sparsity) while improving accuracy on image classification tasks, demonstrating a more efficient and biologically plausible way to design AI.
In the quest to make artificial neural networks (ANNs) more efficient and powerful, researchers often look to the ultimate biological computer: the brain. A new research paper titled “Neuro-inspired Ensemble-to-Ensemble Communication Primitives for Sparse and Efficient ANNs” by Orestis Konstantaropoulos, Stelios Manolis Smyrnakis, and Maria Papadopouli introduces G2GNet, a novel architecture that draws inspiration from the intricate and efficient communication patterns observed in the mouse visual cortex.
The human brain, with its modular, hierarchical, and sparsely interconnected structure, offers a blueprint for designing ANNs that are not only efficient in terms of wiring cost but also specialized and robust. Sparsity, in particular, has been a key area of exploration in deep learning, aiming to reduce memory usage, computational demands, and improve generalization capabilities. This paper delves into how specific patterns of functional connectivity—how groups of neurons communicate with each other—can inform the design of more effective ANNs.
Mimicking Brain Communication
The core inspiration for G2GNet comes from recent findings in systems neuroscience, particularly how neuronal ensembles in the mouse primary visual cortex (V1) transmit information. In the brain, neurons often form “ensembles” that fire synchronously, efficiently transmitting shared information. A key observation is how Layer 4 neurons, which receive primary input, communicate with Layer 2/3 neurons. This communication isn’t random; it’s highly specific and exhibits a “ReLU-like” activation, meaning Layer 2/3 neurons fire strongly only when a significant, but still sparse, number of their Layer 4 partners co-fire. This mechanism promotes reliable information transfer while maintaining sparse activity.
G2GNet translates this biological insight into a structural bias for ANNs. Instead of fully connected layers, G2GNet imposes a sparse, modular connectivity. Each layer is divided into “groups” of neurons. Neurons within a group in one layer are highly likely to connect to neurons in the corresponding group in the next layer (intra-pathway communication). At the same time, there are much sparser connections to neurons in other groups in the next layer (inter-pathway communication). This creates an approximately block-diagonal connectivity matrix, significantly reducing the number of parameters compared to traditional fully connected networks.
Smart Grouping and Dynamic Learning
The way these neuron groups are formed is crucial. The paper explores different “grouping strategies.” An “Index-based” approach groups neurons by their sequential order, preserving spatial locality from the input. A “Random” approach, as the name suggests, assigns neurons to groups arbitrarily. The most effective strategy, termed “Mixer,” alternates between these two. In even-numbered layers, it uses index-based grouping, maintaining spatial coherence. In odd-numbered layers, it uses an interleaved assignment, mixing neurons from different spatial regions. This balance allows G2GNet to process local information while also integrating broader features, leading to better performance.
Beyond this static structural bias, G2GNet also incorporates a “Dynamic Sparse Training” (DST) mechanism. This means the network’s connections aren’t fixed throughout training. Instead, during training, some connections are periodically “pruned” (removed) and an equal number of new connections are “regrown.” This process allows the network to adapt its structure over time, much like synaptic regeneration and rewiring in the brain. A particularly innovative aspect is the introduction of a Hebbian-inspired criterion for this rewiring. This criterion favors connections between neurons whose activations are strongly correlated, aligning with the biological principle that “neurons that fire together, wire together.”
Also Read:
- Enhancing Spiking Neural Networks with Multiple Interacting Learning Mechanisms
- Brain-Inspired Neural Network Design Boosts Efficiency of Equilibrium Propagation
Impressive Results and Future Horizons
The G2GNet architecture has been rigorously tested on standard image classification benchmarks, including Fashion-MNIST, CIFAR-10, and CIFAR-100. Despite having significantly fewer parameters—achieving up to 75% sparsity—G2GNet consistently outperforms dense baselines and even random sparse networks. In some cases, it improved accuracy by up to 4.3% while using a fraction of the computations. The research demonstrates that this biologically inspired structural bias, combined with dynamic sparse training and Hebbian-based rewiring, is highly effective for creating efficient and accurate ANNs.
This work opens exciting avenues for future research. The concept of dynamic group formation in ANNs, driven by input data or internal network states, could lead to even more adaptive and robust models. While the current experiments focus on feedforward layers, the principles of G2GNet are applicable to other architectures, including convolutional layers and the attention mechanisms found in modern transformer models. This research not only provides a powerful new tool for designing efficient ANNs but also serves as a valuable simulation platform for testing neuro-computational hypotheses. You can read the full paper here.
