TLDR: This research paper introduces a new framework for training neural networks that separates the search for diverse solutions from the learning process. It uses evolutionary algorithms to explore a smaller ‘representation space’ (intermediate network activations) to find varied, high-performing solutions. These discovered representations then guide a gradient-based learning phase to train the network’s parameters. The method achieves performance comparable to traditional gradient descent while exploring different solution landscapes, offering a path to overcome gradient descent’s limitations in finding diverse minima.
Neural network training has long relied on gradient descent, a powerful optimization technique that efficiently finds solutions by following the steepest path down a loss landscape. However, this efficiency comes with a significant trade-off: gradient descent typically converges to a single minimum, often missing out on other potentially better-generalizing solutions that might exist elsewhere in the vast parameter space. The challenge lies in the sheer complexity of exploring this high-dimensional space directly.
A new research paper titled DECOUPLING SEARCH AND LEARNING IN NEURAL NET TRAINING by Akshay Vegesna and Samip Dahal from Q Labs proposes an innovative framework to address this fundamental limitation. Their work introduces a method that separates the training process into two distinct phases: an exploratory ‘search’ phase and an efficient ‘learning’ phase.
The Core Idea: Decoupling Search and Learning
The authors argue that the parameter space of modern neural networks, with millions or billions of dimensions, is simply too large for effective direct search. Random sampling or perturbations in this space are highly unlikely to yield good results. Instead, they propose shifting the search to a more tractable domain: the ‘representation space,’ which refers to the intermediate activations within the neural network layers.
This representation space is significantly smaller than the parameter space, making it amenable to search algorithms. Once diverse and high-quality representational solutions are found through search, a separate gradient-based learning phase trains the network’s parameters to produce these discovered representations. This effectively uses search to guide where gradient descent should go, overcoming its inherent exploratory limitations.
Evolutionary Search in Representation Space
The paper details an evolutionary search algorithm that operates directly on these layer-wise activations. Instead of optimizing network parameters, the algorithm evolves the activation tensors at selected layers to minimize classification loss. The process involves initializing a population of noisy input variants and then sequentially evolving representations at each chosen layer, building upon the optimized solutions found for previous layers.
The evolutionary mechanics include selecting top-performing candidates, creating ‘exploratory’ samples with high mutation strength to discover new regions, and ‘refinement’ samples with standard mutation for local improvements. These samples are generated using crossover and Gaussian mutation, with techniques like spatial smoothing and normalization applied to ensure learnability and convergence.
Crucially, the research demonstrates that both the quality (fitness) and diversity of the solutions found by this evolutionary search improve with increased computational resources, such as larger population sizes and more generations. This indicates that the search process can effectively explore and discover a variety of beneficial intermediate representations.
Learning from Discovered Representations
After the evolutionary search identifies the best representations for each training example, these become fixed targets for the subsequent learning phase. The network’s convolutional layers are then trained using gradient descent to match these cached, searched representations through a Mean Squared Error (MSE) objective. A Kullback-Leibler (KL) divergence loss is applied to the final classification head, but with a ‘stop-gradient’ operator that prevents these classification gradients from flowing back into the convolutional layers. This ensures the network’s body learns exclusively from the searched representations.
The authors found that increasing the network’s capacity (e.g., tripling the depth of convolutional blocks) was beneficial for learning these complex searched targets. Their experiments on datasets like MNIST, CIFAR-10, and CIFAR-100 show that this search-based learning approach achieves test accuracies comparable to standard Stochastic Gradient Descent (SGD) training, often within 1% without data augmentation. With data augmentation, a variant that skips direct supervision on the first convolutional block performed even better, narrowing the gap to SGD.
Also Read:
- Neuromorphic Intelligence: A Dynamical Systems Perspective on Brain-Inspired AI
- Neural Cellular Automata: Bridging Biology and AI Through Self-Organizing Systems
Qualitative Differences and Future Directions
A significant finding is that models trained with this decoupled approach are qualitatively different from those trained with SGD. Measurements like cosine distance to the searched targets and collision entropy (which assesses within-class and between-class representation similarity) reveal distinct learning dynamics and representational trajectories. This suggests that the method indeed leads to different types of solutions than traditional gradient descent.
While the paper presents a compelling proof of concept, the authors acknowledge limitations. The performance, though comparable, still slightly trails SGD in some scenarios, indicating a need for further refinement. Additionally, the current approach uses a one-shot search with cached representations. Future work aims to implement tighter feedback loops where trained networks can inform subsequent search iterations, creating a more dynamic and iterative optimization cycle. This research opens exciting avenues for developing new training algorithms that combine the exploratory power of search with the efficiency of gradient-based learning.
