TLDR: GENESIS is a novel computational model that unifies semantic and episodic memory interactions. It uses a Cortical-VAE for semantic learning and a Hippocampal-VAE within a Retrieval-Augmented Generation (RAG) system for episodic memory. The model successfully reproduces key memory phenomena like generalization, recognition, serial recall, gist-based distortions, and constructive episodic simulation, highlighting how limited capacity shapes memory fidelity and retrieval.
A fundamental question in understanding the human mind is how our different types of memory work together. Specifically, how do semantic memory (our knowledge of facts and concepts) and episodic memory (our personal experiences) interact? For a long time, computational models have struggled to provide a unified framework that explains how these two major forms of declarative memory, often linked to the cortex and hippocampus respectively, support learning, recall, and even imagination.
Addressing this challenge, a new computational model called the Generative Episodic–Semantic Integration System, or GENESIS, has been introduced. This model offers a fresh perspective on how semantic and episodic systems interact, while also successfully reproducing key findings from both memory domains. GENESIS views memory as a dynamic interplay between two generative systems with limited capacity: a Cortical-VAE, which handles semantic learning and generalization, and a Hippocampal-VAE, responsible for episodic encoding and retrieval within a Retrieval-Augmented Generation (RAG) architecture.
Understanding GENESIS: The Core Components
At its heart, GENESIS consists of three main parts: the Cortical-VAE, the Hippocampal-VAE, and an Episodic Memory component. Both VAEs (Variational Autoencoders) are generative models designed with limited capacity, meaning they compress information, much like our brains do when faced with vast amounts of data.
- Cortical-VAE: This component acts like the brain’s cortex, learning structured knowledge and generalizing from experiences. When an input, like an image of a red ‘6’, is presented, the Cortical-VAE encodes it into a compressed ‘item embedding’ that captures its features (like color and digit identity). This embedding can then be decoded to reconstruct the original perception.
- Hippocampal-VAE: This acts like the hippocampus, further compressing the item embedding from the Cortical-VAE and combining it with a ‘temporal embedding’ that marks when the experience occurred. This compressed representation forms a ‘key’ for the memory.
- Episodic Memory (RAG System): This is where memories are stored as ‘key-value pairs’. The ‘key’ is the compressed representation from the Hippocampal-VAE (item + temporal embedding), and the ‘value’ is the more detailed item embedding from the Cortical-VAE. Retrieval works by matching a ‘query’ (e.g., a probe image or a temporal cue) to these stored keys.
How GENESIS Explains Memory Phenomena
The GENESIS model has been shown to replicate several hallmark behavioral findings across both semantic and episodic memory:
Semantic Learning and Generalization: The Cortical-VAE demonstrates how semantic memory learns statistical regularities from inputs and can generalize this knowledge to create novel combinations (e.g., if it learns ‘3-red’ and ‘5-blue’, it can generate ‘5-red’). The model also shows that limited capacity in the Cortical-VAE leads to a decline in reconstruction accuracy, much like how our attention or cognitive resources can affect how well we process information.
Recognition Memory: GENESIS successfully simulates ‘old/new’ recognition tasks. When presented with a list of items and then probed, the model recognizes previously seen items based on how well the probe’s embedding matches stored keys. Crucially, recognition accuracy decreases with longer lists and lower Hippocampal-VAE capacity, reflecting how more memories or less distinct encoding makes retrieval harder.
Serial Recall: By incorporating temporal embeddings into the memory keys, GENESIS reproduces serial recall effects. Items are recalled in their original order, and the model shows a ‘recency effect’ (better recall of recent items) that diminishes with delayed recall. The model also illustrates how a stronger emphasis on temporal information improves ordered recall, while a weaker emphasis can lead to ‘semantic intrusions’ – recalling semantically related but incorrect items.
Gist-based Distortions: A fascinating aspect of human memory is how details can be lost, and memories can become biased towards a ‘gist’ or prototype. GENESIS demonstrates this by showing that when the Cortical-VAE encodes memories with low capacity, fine-grained details are lost. When these memories are later reconstructed, they become more similar to a general prototype of the category rather than the specific original experience.
Constructive Episodic Simulation and Recombination: The model provides a mechanism for how we can recombine past experiences to imagine novel scenarios, a process known as constructive episodic simulation. By combining different components from stored item embeddings (e.g., the digit from one memory, the color from another), GENESIS can generate plausible images of previously unexperienced combinations. This highlights how memory is not just about replaying exact events but actively constructing new ones.
Also Read:
- AUGUSTUS: An AI Agent with Human-Like Multimodal Memory
- Rethinking AGI: Why Theory, Not Just Data, Holds the Key to General Intelligence
A Unified Framework for Memory
GENESIS stands out from traditional models like the Complementary Learning Systems (CLS) theory by proposing a more integrated and bidirectional interaction between semantic and episodic memory. Instead of episodic memories being formed independently, the semantic system (Cortical-VAE) is integral to both encoding and decoding episodic representations. Furthermore, memories are stored as latent embeddings rather than raw perceptual items, aligning with theories that the hippocampus indexes cortical representations rather than storing full content.
The model also emphasizes the role of limited capacity in both cortical and hippocampal processing, offering a normative explanation for why memories lose detail or become harder to retrieve. This work provides a principled account of memory as an active, constructive, and resource-bounded process, offering new insights into the generative foundations of human cognition. For more in-depth information, you can read the full research paper here.
