TLDR: OptiProxy-NAS is a new framework for Neural Architecture Search (NAS) that transforms the complex, discrete search space into a continuous, differentiable, and smooth one using an “optimization proxy.” This allows for efficient, gradient-based optimization, leading to superior performance and significantly fewer computational queries compared to existing methods across various tasks, including computer vision, natural language processing, and hardware-aware scenarios. It also offers flexibility in incorporating diverse real-world metrics.
Designing high-performing neural networks, a process known as Neural Architecture Search (NAS), is a critical but incredibly challenging task in artificial intelligence. Imagine trying to find the best possible blueprint for a complex machine from an almost infinite number of possibilities – that’s the scale of the problem. This process is not only computationally expensive but also navigates a vast, discrete, and often unpredictable search space, making it notoriously difficult to find optimal solutions efficiently.
Current approaches to NAS often fall into two main categories, each with its own set of limitations. One popular method involves using “predictor-based” techniques, where a surrogate model tries to predict the performance of a neural network without fully training it. While this can speed things up, these predictors can be unreliable, biased, and don’t always perform consistently across different tasks. They still require a significant amount of actual network training to guide the search.
Another prominent strategy is “differentiable NAS,” which uses a large “supernetwork” that encompasses many possible architectures. By making the architectural choices differentiable, researchers can use gradient-based optimization to find the best sub-network. However, this method often struggles with incorporating real-world, non-differentiable metrics like power consumption or latency, and can demand a lot of memory.
A new research paper titled “OptiProxy-NAS: Optimization Proxy based End-to-End Neural Architecture Search” by Bo Lyu, Yu Cui, Tuo Shi, and Ke Li introduces a novel framework to overcome these hurdles. The core idea behind OptiProxy-NAS is to redefine the NAS problem as an end-to-end optimization challenge by introducing an “optimization proxy.” This proxy transforms the typically discrete and complex search space into one that is continuous, differentiable, and smooth. This transformation is crucial because it allows the application of powerful, gradient-based optimization methods, which are far more efficient than traditional search techniques.
The OptiProxy-NAS framework offers several key advantages. Firstly, it significantly boosts search efficiency by enabling a direct flow of gradient information between the architectural choices and the performance metrics. Unlike previous methods where the search algorithm and performance evaluation were somewhat isolated, OptiProxy-NAS integrates them seamlessly. This means the system can learn more effectively how to adjust architectural parameters to improve performance.
Secondly, the framework is remarkably flexible. It can directly incorporate a wide array of real-world metrics beyond just accuracy, such as latency, power consumption, and other resource constraints. This is a major step forward, especially for designing neural networks for specific hardware or low-resource environments. It achieves this without the need for costly supernetworks, making it more practical for deployment-aware NAS scenarios.
Furthermore, OptiProxy-NAS progressively models the NAS landscape. Instead of trying to represent the entire vast space at once, it starts with a coarse estimation and refines its understanding of promising regions as the search progresses. This adaptive approach helps in efficiently navigating the search space and finding high-performing architectures.
The researchers conducted extensive experiments across 12 NAS tasks on 4 different search spaces, covering diverse domains like computer vision, natural language processing, and resource-constrained NAS. The results consistently demonstrated the superior search performance and efficiency of OptiProxy-NAS compared to 15 state-of-the-art algorithms. For instance, on the NAS-Bench-201 benchmark, OptiProxy-NAS achieved optimal architectures with nearly 100% fewer queries for CIFAR-10 and CIFAR-100 datasets. It also showed significant efficiency improvements (up to 400%) on NAS-Bench-301 and NAS-Bench-NLP, and outperformed existing methods in hardware-aware tasks with 50% fewer queries.
Even in low-fidelity scenarios, where only a few training epochs were allowed, OptiProxy-NAS proved its compatibility and flexibility, achieving satisfying results. The additional computational cost introduced by the optimization proxy is minimal, adding only about 20 seconds and 10MB of memory, which is a tiny fraction of the total NAS pipeline resources.
Also Read:
- LM-Searcher: Unifying Neural Architecture Search Across Domains Using LLMs
- BranchGRPO: A New Approach for Stable and Fast Generative Model Alignment
In conclusion, OptiProxy-NAS represents a significant advancement in the field of Neural Architecture Search. By reformulating NAS as a differentiable optimization problem through an innovative optimization proxy, it offers a more efficient, flexible, and effective way to design neural networks. This framework not only achieves superior results with fewer computational resources but also opens new avenues for integrating diverse real-world constraints directly into the architecture search process. For more in-depth technical details, you can refer to the full research paper here.
