TLDR: This paper introduces WRCor, a new training-free method for Neural Architecture Search (NAS) that efficiently evaluates neural networks without training. WRCor measures a network’s expressivity and generalizability by analyzing the correlation of activations and gradients across different layers and inputs, with a focus on top-layer responses. Combined with voting proxies like SJW, this approach significantly speeds up NAS, enabling the discovery of high-performing architectures (e.g., 22.1% test error on ImageNet-1k) in just four GPU hours, outperforming many existing NAS algorithms.
Neural Architecture Search, or NAS, is a powerful method for automatically designing neural network architectures. Traditionally, this process has been incredibly resource-intensive and time-consuming, often requiring the training of many different network designs from scratch. This computational burden has been a significant hurdle in the widespread adoption of NAS.
To address this, researchers have developed ‘zero-shot’ NAS methods. These innovative approaches aim to estimate the performance of neural architectures without the need for extensive training. While zero-shot methods offer a significant leap in efficiency, existing techniques have often fallen short in terms of consistent effectiveness, stability, and broad applicability across different neural network designs.
A new research paper introduces a novel training-free estimation method called Weighted Response Correlation, or WRCor. This method offers a fresh perspective on evaluating neural network architectures. WRCor works by analyzing the correlation coefficient matrices of ‘responses’ – which include both activations and gradients – across different input samples. By doing so, it calculates a ‘proxy score’ that effectively measures two crucial aspects of a neural network: its ‘expressivity’ and its ‘generalizability’.
Understanding Expressivity and Generalizability
Expressivity refers to a neural network’s ability to distinguish between different inputs. The paper suggests that networks with greater expressivity will have activations that are more linearly independent across different samples. In simpler terms, if a network can produce distinct and uncorrelated internal representations for different inputs, it’s better at telling them apart.
Generalizability, on the other hand, relates to how well a neural network performs on new, unseen data after it has been trained. The research posits that networks with better generalizability will exhibit lower correlation in their gradients across different inputs. This implies that the network’s learning adjustments are not overly aligned, allowing for a broader exploration of solutions during training and thus better performance on diverse data.
How WRCor Works
WRCor combines these two concepts. It computes correlation matrices for both activations and gradients. A key insight of the paper is that for superior neural architectures, the correlations of responses in their top layers are often greater than those in their bottom layers. This means that while bottom layers might capture common features, top layers are responsible for more independent, discriminative features. To reflect this, WRCor assigns higher weights to the correlation matrices from the top layers of the network when calculating the overall proxy score, giving them more influence in the evaluation.
This approach offers several advantages: it can be applied to any neural architecture, it considers both expressivity and generalizability, it simplifies the calculation process, and it accounts for the varying importance of different layers.
Enhancing Performance with Voting Proxies
Recognizing that no single evaluation method is perfect for all scenarios, the researchers also propose ‘voting proxies’ to further enhance performance and stability. These voting proxies combine WRCor with other strong existing proxies. One such voting proxy, named SJW, leverages SynFlow, JacCor, and WRCor. Another, SPW, uses SynFlow, PNorm, and WRCor. These combinations aim to balance the strengths of different evaluation metrics, especially in challenging situations like evaluating top-ranked architectures or exploring very large search spaces.
Also Read:
- A New Approach to Combat Catastrophic Forgetting in AI: Focusing on the Final Learning Stages
- Boosting LoRA Performance: A Constraint-Based Approach to Adapter Initialization
Experimental Success
The experimental results are compelling. In proxy evaluations, WRCor and its voting proxies demonstrated superior efficiency, stability, and generality compared to many existing methods. When applied to actual architecture search, the zero-shot NAS algorithms using WRCor and SJW consistently outperformed most existing NAS algorithms in image recognition tasks.
Notably, an algorithm called RE-SJW, which combines regularized evolution with the SJW voting proxy, achieved a highly competitive result of 22.1% top-1 test error on the challenging ImageNet-1k dataset. What’s truly remarkable is that this was accomplished within just four GPU hours, making it significantly more efficient than many other state-of-the-art NAS algorithms. This efficiency allows for the discovery of high-performing architectures with substantially less computational cost.
This research marks a significant step forward in making Neural Architecture Search more accessible and efficient, paving the way for faster discovery of advanced neural network designs. For more details, you can refer to the full research paper: Zero-Shot Neural Architecture Search with Weighted Response Correlation.
