TLDR: Budgeted Broadcast (BB) is a novel neural network pruning method inspired by biological metabolic constraints. It defines a neuron’s ‘traffic’ as the product of its activity and fan-out, enforcing a local budget to prune connections. This approach promotes a ‘selectivity-audience balance,’ protecting rare but important features while improving overall efficiency and accuracy. BB has shown superior performance in tasks like speech recognition, face identification, change detection, and synapse prediction, often surpassing dense models and overcoming optimization challenges.
Modern deep neural networks, while powerful, often learn highly redundant representations and can struggle with rare events. This inefficiency stems from their training with abundant computational resources, a stark contrast to biological neural circuits that operate under strict metabolic constraints. A new research paper introduces a novel approach called Budgeted Broadcast (BB) that aims to instill a principle of biological resource efficiency into artificial neural networks, making them more robust and diverse.
Traditional pruning methods primarily focus on a neuron’s ‘utility’ – its importance based on weight magnitude or gradient information. However, Budgeted Broadcast introduces an orthogonal concept: a neuron’s ‘cost,’ defined as the resources it consumes to broadcast its signal. This cost is formalized as ‘traffic,’ which is the product of a neuron’s long-term activity (how often it ‘speaks’) and its fan-out (the size of its ‘audience’ or outgoing connections).
Inspired by formal models of metabolic constraints in neuroscience, BB enforces a local budget on this traffic. If a unit’s traffic exceeds a predefined threshold, it prunes its weakest connections. This can involve reducing its fan-out (axonal pruning) or reducing its fan-in (dendritic pruning) to lower its activity. This mechanism creates a crucial trade-off: neurons can either ‘speak’ loudly to a small audience or quietly to a large one, but not both. Crucially, this protects highly selective, rare-feature detectors (which have low activity) by treating them as metabolically cheap, allowing them to maintain a larger audience.
This simple, local rule leads to a global organizing principle. An analysis of the network’s coding entropy predicts that this budget pressure drives the network to self-organize into a measurable equilibrium called ‘selectivity–audience balance.’ This balance suggests a linear relationship between a unit’s fan-out and its inactivity log-odds, linking a unit’s structure to its function.
The Budgeted Broadcast method offers several significant contributions. It formalizes a traffic budget, deriving a testable selectivity-audience balance as the system’s equilibrium state. Implemented as a minimal, local controller, BB has been validated on controlled tasks, showing it produces the predicted balance, provides structural safety for rare-but-relevant signals, and overcomes optimization barriers that can stall standard gradient-based methods.
The utility of BB has been demonstrated across four diverse domains. In Automatic Speech Recognition (ASR) on LibriSpeech, BB consistently improved Word Error Rate (WER), especially for rare words, outperforming other methods. For face identification using a ResNet-101, BB formed or matched the upper envelope of performance across a broad range of sparsity levels, often exceeding dense baselines. In change detection on the LEVIR-CD dataset, BB improved Intersection-over-Union (IoU) and F1-score, recovering substantially more true positives for small, scattered changes. Finally, for synapse prediction in 3D U-Nets on electron microscopy images, BB attained state-of-the-art F1 and PR-AUC scores.
Also Read:
- PrunedLoRA: Enhancing Low-Rank Adaptation for Large Language Models Through Dynamic Pruning
- Compressing Neural Networks for Efficient AI Deployment
Budgeted Broadcast introduces a new perspective on structural plasticity in artificial neural networks, shifting the focus from a component’s utility to its metabolic cost. This computational framework provides a unified explanation for biological phenomena like Henneman’s size principle and synapse elimination, reframing them as solutions to efficient information broadcast. The success of BB across various benchmarks provides strong empirical support for this structural perspective of neural organization. For more in-depth information, you can read the full research paper here.
