TLDR: This research paper introduces a novel multi-agent reinforcement learning framework inspired by biological inclusive fitness. Agents are assigned genotypes, and their rewards are modified to account for genetic similarity with other agents, promoting a spectrum of cooperative behaviors. Experiments in network games demonstrate that this ‘inclusive reward’ function leads to the emergence of cooperation consistent with biological principles like Hamilton’s rule. The paper outlines future work in open-ended environments like Neural MMO, proposing different evolutionary aligned reward functions (longevity, replication, combined) to foster complex, non-team-based social dynamics and a continuous ‘autocurriculum’ of strategic development, aiming to create more adaptable and socially intelligent AI agents.
Artificial intelligence has made incredible strides, particularly in single-agent reinforcement learning where agents master specific tasks. However, a significant challenge remains: creating truly intelligent agents that can adapt to a wide variety of complex situations, much like living organisms in nature. Traditional methods often hit a ceiling, requiring extensive manual effort to design new tasks or reward signals, or getting stuck in suboptimal learning paths.
Inspired by the millions of years of evolution that have shaped the complex social behaviors and intelligence we see in nature, a new research paper proposes a novel approach to multi-agent reinforcement learning. The paper, titled “Inclusive Fitness as a Key Step Towards More Advanced Social Behaviors in Multi-Agent Reinforcement Learning Settings,” introduces a framework where agents are designed with a concept borrowed directly from biology: inclusive fitness.
The core idea revolves around assigning each agent a ‘genotype’ – a unique sequence of abstract genes. These genes can be shared among different agents. To quantify how genetically similar two agents are, the researchers use a metric called ‘Hamming similarity,’ which essentially measures how many genes they have in common. This genetic similarity then plays a crucial role in how agents receive their rewards.
In this framework, an agent’s reward isn’t just based on its own performance. It’s an ‘inclusive reward’ that also considers the rewards of other agents, weighted by their genetic relatedness. This means that helping a genetically similar agent contributes to an agent’s own inclusive reward. This mechanism naturally encourages a spectrum of cooperative behaviors, moving beyond the simple binary choices of full competition or full cooperation often seen in previous multi-agent AI research.
The researchers, Andries Rosseau, Raphaël Avalos, and Ann Nowé from Vrije Universiteit Brussel, tested their inclusive reward function in network games, specifically variations of the classic Prisoner’s Dilemma. In these experiments, agents learned to cooperate with others based on their genetic similarity, aligning with well-established biological principles like Hamilton’s rule. They observed that cooperation increased significantly when inclusive rewards were used, especially in scenarios where agents were more likely to interact with genetic relatives, mimicking ‘limited dispersal’ in natural populations.
Looking ahead, the team plans to extend this framework to more complex, open-ended environments, such as the Neural MMO platform. This environment is a multi-agent video game where agents must survive by gathering resources, engaging in combat, and crucially, reproducing. In these dynamic settings, the concept of fitness becomes more intricate, leading to different types of ‘evolutionary aligned rewards’:
-
Longevity Reward
Agents are rewarded for every time step at least one copy of their unique genotype is alive, with additional rewards for related genotypes.
-
Replication Reward
Agents receive rewards for newborns and penalties for deaths, again weighted by genetic similarity, promoting the spread of their genetic material.
- Also Read:
- AnimaRL: A Data-Driven Simulator for Multi-Animal Behavior with Unknown Dynamics
- Closing Theoretical Gaps: New Algorithm Achieves Optimal Efficiency in Multi-Agent Imitation Learning
Combined Reward
A blend of the above, giving a positive reward (based on Hamming similarity) for every agent carrying the same genetic material that remains alive for another time step, also accounting for the number of copies.
In such environments, the researchers hypothesize the emergence of an ‘arms race of strategies,’ where agents continuously improve, pushing others to adapt, creating a multi-agent autocurriculum analogous to biological evolution. As resources become scarce, the framework is expected to foster complex social dynamics, where agents might form non-team-based coalitions, mediate conflicts, or even strategically eliminate distant relatives to favor closer ones. This could lead to highly sophisticated and nuanced social intelligence.
The training of these agents will utilize advanced Deep Reinforcement Learning techniques like Proximal Policy Optimization (PPO) and Long Short-Term Memory (LSTM) networks. The world itself will evolve organically, starting with a single agent, with offspring inheriting genotypes that can mutate, leading to new ‘species’ and strategies over time.
This research offers a promising path toward creating AI agents that are not only capable but also possess a deeper, more adaptive form of social intelligence, driven by simple, biologically inspired rules. For more details, you can read the full research paper here.
