TLDR: Chimera is a novel framework that uses multi-agent large language models (LLMs) to automatically simulate realistic insider threats and benign employee activities within various enterprise environments. It generates a high-quality, large-scale dataset called ChimeraLog, which includes diverse log modalities and attack scenarios. Evaluations show ChimeraLog is highly realistic and more challenging for existing insider threat detection methods, addressing the critical data scarcity bottleneck in cybersecurity research.
Insider threats, which are security incidents originating from within an organization, pose a significant and costly challenge in today’s digital landscape. These threats, often perpetrated by trusted individuals with legitimate access, are notoriously difficult to detect. A major hurdle in developing effective detection methods is the scarcity of high-quality, real-world data. Existing public datasets are either limited in scope, lack rich semantic information, or are too sensitive to be shared.
Addressing this critical data gap, researchers have introduced a groundbreaking framework called Chimera. This is the first large language model (LLM)-based multi-agent system designed to automatically simulate both normal employee activities and malicious insider behaviors, while also collecting detailed logs across various enterprise environments.
Chimera operates by customizing each LLM agent to represent an individual employee, complete with detailed role modeling, and integrates modules for group meetings, one-on-one interactions, and self-organized scheduling. This sophisticated design allows Chimera to accurately reflect the complexities of real-world enterprise operations. The current version of Chimera can simulate 15 distinct types of insider attacks, including intellectual property theft and system sabotage.
To demonstrate its capabilities, Chimera was deployed to simulate benign and attack activities across three common data-sensitive organizational scenarios: a technology company, a finance corporation, and a medical institution. Over a one-month simulation period, this process generated a new, extensive dataset named ChimeraLog. This dataset is designed to significantly advance the development of machine learning-based insider threat detection (ITD) methods.
ChimeraLog is remarkable for its size and diversity, comprising approximately 20 billion normal log entries and 5 billion attack log entries. It includes six distinct log modalities: four application-level logs (login information, email communications, web browsing history, and file operation records) and two system-level logs (network traffic and system logs). This makes ChimeraLog the largest and most diverse ITD dataset available to date, offering a rich context for analyzing both normal workflows and malicious behaviors.
Comprehensive evaluations, including human studies with security experts and quantitative analyses, have confirmed the quality and authenticity of ChimeraLog. Experts rated ChimeraLog as highly realistic, comparable to real-world datasets, and significantly more realistic than older synthetic datasets. Furthermore, existing insider threat detection methods found ChimeraLog more challenging to analyze, with an average F1-score of 0.83, notably lower than the 0.99 observed on a baseline dataset. This highlights the increased difficulty and realism posed by ChimeraLog for threat detection tasks.
The research also explored the impact of different foundation models (like OpenAI GPT-4o, Google Gemini-2.0-Flash, and DeepSeek V3) on the quality of the generated data. While all models produced effective data for training detection models, GPT-4o facilitated the most diverse and communicative log outputs, leading to higher volumes of events and richer insider interactions.
Also Read:
- A Dual-Modality AI Framework for Enhanced Insider Threat Detection
- Unmasking Insider Threats Through Behavioral Signal Analysis
This work offers promising insights for the future of cybersecurity. The success of Chimera underscores the potential of LLM-based multi-agent frameworks for automated applications in software engineering and security domains. It also highlights the critical impact of ‘distribution shift’ on ITD model performance, emphasizing the need for robust methods that can adapt to evolving threat landscapes. For more details on this innovative framework, you can refer to the full research paper here.
