TLDR: A new research paper explores how Developmental Graph Cellular Automata (DGCAs), a model inspired by biological morphogenesis, can be used to ‘grow’ neural network reservoirs. These DGCAs are trained using a genetic algorithm to develop complex, ‘life-like’ graph structures that can effectively solve computational tasks, often outperforming randomly initialized networks. The study highlights the potential for creating ‘plastic’ reservoirs that can adapt and self-repair, paving the way for more functional and adaptive artificial intelligence systems.
Researchers Matias Barandiaran and James Stovold from Lancaster University Leipzig have introduced a novel approach to developing neural networks, specifically focusing on a type of network called ‘reservoirs’. Their work, detailed in the paper Growing Reservoirs with Developmental Graph Cellular Automata, explores how these computational systems can be ‘grown’ rather than traditionally designed, drawing inspiration from biological morphogenesis.
Understanding the Challenge of Neural Network Design
Traditional recurrent neural networks (RNNs), while powerful for sequential data like time series or language, are computationally intensive to train. Reservoir Computing (RC) offers a solution by using a large, fixed, randomly connected RNN (the ‘reservoir’) where only a small ‘readout’ layer is trained. This simplifies the process significantly. A fascinating aspect of RC is its ability to leverage ‘physical reservoirs’ – actual physical systems like biological neurons or nanomaterials – for computation, offering benefits like low power consumption and fault tolerance.
However, conventional RC often relies on trial-and-error to find suitable reservoirs. For physical systems, actively designing or adapting the substrate is even harder. This led the researchers to a fundamental question: could Developmental Graph Cellular Automata (DGCAs) be used to grow these reservoirs?
Introducing Developmental Graph Cellular Automata (DGCAs)
DGCAs are an extension of Neural Cellular Automata (NCAs), which are models for morphogenesis capable of growing predefined shapes from a single ‘seed’ cell. Unlike NCAs, which operate on a grid, DGCAs allow for the growth of directed graphs, better mimicking the complexity of biological environments. These graphs start from a single node and expand over time, with each node making local decisions about division, removal, or stasis based on its own and neighboring states. This distributed control allows for the emergence of complex structures.
Growing Reservoirs with a Genetic Algorithm
Since the growth process of DGCAs is non-differentiable, traditional optimization methods like backpropagation cannot be used. Instead, the researchers employed a variant of the Microbial Genetic Algorithm (MGA) to search for optimal neural network weights. In this evolutionary process, different ‘individuals’ (reservoirs) compete, with the ‘loser’ adopting some of the ‘winner’s’ genetic material and undergoing mutations. The graphs grown by the DGCA were also made more complex by allowing edge weights to be positive or negative (bipolarization) and nodes to represent different types of neurons (e.g., linear or hyperbolic tangent).
Experiments and Key Findings
The study conducted two main types of experiments:
-
Task-Driven Growth: Reservoirs were grown to solve NARMA (Normalized Auto-Regressive Moving Average) tasks, which are benchmarks for imitating open systems. The results showed that DGCA-grown reservoirs consistently outperformed randomly initialized ‘typical’ reservoirs, even with fewer neurons. Interestingly, the DGCAs naturally grew into a variety of specialized, ‘life-like’ structures, including ‘Linear’, ‘Loosely Stranded’, and ‘Other’ categories. Loosely Stranded graphs often performed best, especially under tighter resource constraints. As task difficulty increased, the DGCAs specialized in ways that prioritized metrics like Generalization Rank (GR) and Linear Memory Capacity (LMC), even without explicit optimization for them.
-
Task-Independent Growth: Here, reservoirs were grown by optimizing for widely accepted Reservoir Computing metrics (Kernel Rank, Generalization Rank, Linear Memory Capacity, and Spectral Radius) rather than direct task performance. While task-specific growth still yielded better results, reservoirs grown solely based on these metrics still performed well on NARMA tasks, suggesting these metrics are reasonable proxies for reservoir dynamics.
Also Read:
- Giving Artificial Organisms an ‘Identity’ Boosts Stability and Promotes Emergent Movement
- Unlocking Lifelong Learning: The Rise of Self-Evolving AI Agents
Implications and Future Directions
This research demonstrates that DGCAs can effectively grow functional reservoirs, producing diverse and specialized graph configurations that can solve benchmark tasks. The emergence of low-density, linear strand topologies as high-performing solutions suggests that even sparse networks can be effective. A significant implication is the potential for developing ‘plastic reservoirs’ – systems that can adapt to environmental changes, new tasks, or even self-repair after damage, a capability lacking in traditional fixed-structure reservoirs.
Future work will explore integrating Liquid State Machines, expanding the task domain, and addressing the current limitation of slow training due to the reliance on search-based methods. The researchers also aim to develop input-output schemes inspired by physical reservoir computing, where the reservoir grows around fixed input and output neurons.
Overall, this study lays a strong foundation for using DGCAs as a viable framework for reservoir computing, opening new avenues for exploring growth mechanisms inspired by nature’s purposeful development of complex systems.
