TLDR: AUGUSTUS is a novel multimodal AI agent system inspired by human memory, capable of processing, storing, retrieving, and acting on text, image, audio, and video data. It introduces a graph-structured ‘contextual memory’ that uses semantic tags for efficient, concept-driven retrieval via its ‘CoPe search’ algorithm. The system demonstrates superior performance and efficiency compared to traditional multimodal RAG and MemGPT, marking a significant advancement in AI agents with human-like memory capabilities.
In the rapidly evolving landscape of artificial intelligence, large language models (LLMs) have shown remarkable capabilities, especially when augmented with external memory systems. However, a significant gap has persisted: most existing AI agents primarily focus on storing and retrieving text-based information, largely overlooking the rich, multimodal nature of human memory.
A new research paper introduces AUGUSTUS, an innovative multimodal agent system that draws inspiration directly from the principles of human memory in cognitive science. Developed by Jitesh Jain, Shubham Maheshwari, Ning Yu, Wen-mei Hwu, and Humphrey Shi, AUGUSTUS aims to bridge this gap by enabling AI agents to process, store, and retrieve information across various modalities, including text, images, audio, and video.
Mimicking Human Cognition
The core design of AUGUSTUS is aligned with the human cognition system, operating through a continuous loop of four stages: encode, store in memory, retrieve, and act. This structure allows the agent to understand inputs, save important information, search for relevant context, and then perform tasks accordingly.
- Encode: AUGUSTUS leverages a suite of foundation models to understand multimodal inputs from users. All information, regardless of its original modality, is converted into a language form for the LLM to process and store.
- Store in Memory: The system employs three types of memory. An ‘in-context memory’ acts like human working memory, holding limited, immediately accessible information. A ‘recall memory’ logs the complete conversation history in a text database. Most notably, a ‘contextual memory’ serves as a hierarchical, multimodal database, storing user context by abstracting information into semantic tags (concepts) and associating them with their original multimodal content.
- Retrieve: AUGUSTUS autonomously decides when to search for information. For contextual memory, it uses a novel two-stage ‘Contextual-Personalized (CoPe) search’ algorithm. This method first searches for relevant semantic tags and then retrieves associated context nodes, significantly reducing search space and improving efficiency compared to traditional sequential databases.
- Act: The agent is equipped with a range of actions, including sending messages, generating images, audio, and video, and even editing images, allowing for diverse and personalized responses.
The Power of Contextual Memory and CoPe Search
Unlike conventional systems that rely on sequential key-value databases, AUGUSTUS’s contextual memory is organized in a graph structure. Information is conceptualized into ‘tag nodes’ (semantic concepts) which are then connected to ‘context nodes’ that store the actual multimodal data (text, image, video, audio) from conversation snapshots. This graph-based approach allows for efficient, concept-driven retrieval, as associations among tags are based on shared context.
The CoPe search algorithm is a cornerstone of AUGUSTUS’s efficiency. By first searching over a contextual tree of tags and then leveraging a personalized knowledge graph to predict additional relevant tags, the system can quickly pinpoint and retrieve highly relevant, personalized information. This method has been shown to be significantly faster than traditional multimodal Retrieval-Augmented Generation (RAG) approaches, especially with large memory sizes.
Performance and Efficiency
Empirical evidence demonstrates AUGUSTUS’s effectiveness. On the ImageNet-1k classification task, its CoPe search mechanism showed impressive retrieval performance, comparable to specialized classification frameworks. Crucially, it outperformed traditional multimodal RAG by being nearly 3.5 times faster for the same memory size, highlighting the efficiency of its concept-driven search.
Furthermore, AUGUSTUS proved its mettle as a conversational agent on the Multi-Session Chat (MSC) benchmark, outperforming other methods like MemGPT in maintaining conversation consistency and answering questions based on prior user conversations. This indicates its superior ability to remember and leverage user context over extended interactions.
Also Read:
- AI Agents Learn and Adapt Through Dialogue to Tackle Complex Problems
- Upgrading Multimodal AI Data: The VERITAS Pipeline
An Open-Source Future for AI Agents
The researchers prioritized using open-source models for AUGUSTUS’s tools, aiming for better reproducibility and lower financial costs, though the system also supports API-based models. This commitment to open-source tools, combined with its innovative memory architecture, positions AUGUSTUS as a significant step towards developing more dependable and cognition-aligned AI agent systems.
For more in-depth information, you can read the full research paper here.
