TLDR: ForensicsData is a new, publicly available Question-Context-Answer (Q-C-A) dataset comprising over 5,000 triplets, created from 2025 malware analysis reports using various Large Language Models (LLMs). It addresses the critical scarcity of realistic digital forensics datasets, enabling better training and evaluation for forensic tools. The dataset was generated through a scalable pipeline involving data collection, preprocessing, LLM-based annotation, and a multi-layered validation process, ensuring high quality, semantic diversity, and logical consistency.
Digital forensics is a crucial field in cybersecurity, focusing on collecting, preserving, examining, and analyzing digital evidence for legal proceedings and incident response. However, investigators face significant challenges due to the sheer volume of diverse data and, more critically, the scarcity of realistic, publicly available datasets for training and testing forensic tools. This limitation often stems from strict privacy regulations, legal restrictions, and the sensitive nature of forensic evidence, particularly in areas like malware analysis where threats constantly evolve.
The emergence of Large Language Models (LLMs) offers a promising solution to this problem. LLMs, built on advanced AI architectures, can understand and generate natural language with remarkable proficiency. In digital forensics, they show potential for automating evidence triage, streamlining malware analysis, and generating comprehensive investigative reports. A particularly compelling application is their ability to create realistic synthetic datasets, which can mimic authentic forensic data without the privacy and legal constraints of real-world evidence.
Introducing ForensicsData: A Novel Dataset for Digital Forensics
To address the critical gap in available datasets, researchers have introduced ForensicsData, an extensive Question-Context-Answer (Q-C-A) dataset specifically designed for digital forensics. This dataset comprises over 5,000 Q-C-A triplets, all sourced from actual malware analysis reports published in 2025. It is the first publicly available, structured Q-C-A resource tailored for digital forensics applications, providing valuable information on malware metadata, behavioral patterns, indicators of compromise (IOCs), tactics, techniques, and procedures (TTPs), and mitigation strategies.
How ForensicsData Was Created
The creation of ForensicsData involved a unique and rigorous workflow. The process began with collecting 1,500 malware execution reports from ANY.RUN, a public malware analysis platform. These reports, all from 2025, covered 15 diverse malware families (e.g., Remote Access Trojans, ransomware, credential stealers) and included benign samples for comparative analysis. After collection, the raw HTML and XML reports underwent extensive preprocessing, including parsing, noise removal, standardization into JSON format, and deduplication to ensure a clean and structured dataset.
The core of the dataset generation involved a scalable LLM annotation pipeline. Structured JSON reports were fed into five different LLMs: Mistral 8B, LLaMA 3–70B, DeepSeek V3, Qwen-QWQ-32B, and Gemini 2.0 Flash. Each LLM processed a portion of the dataset, generating five Q-C-A triplets per report based on carefully engineered prompts. These prompts were designed to extract diverse forensic insights across five key dimensions: malware identification, technical indicators, suspicious and malicious behavioral patterns, and impact assessment/mitigation strategies. This cross-model approach enhanced the stylistic and conceptual variety of the dataset.
Ensuring Quality and Reliability
Given the synthetic nature of the dataset, a multi-layered validation strategy was crucial to ensure its quality, coherence, and utility. This included:
- Format Validation: Using the Pydantic library, each Q-C-A entry was checked for schema correctness and structural consistency. Over 98% of entries passed this validation.
- Deduplication and Similarity Filtering: Vector-based similarity analysis, employing the all-MiniLM-L6-v2 Embedding Model, identified and removed semantically overlapping questions, ensuring high semantic diversity.
- LLM-as-Judge Evaluation: A high-performing LLM, Gemini 2.0 Flash, acted as an automated evaluator. It assessed each Q-C-A triplet against four criteria: logical validity, relevance, completeness, and consistency/absence of hallucinations. This rigorous check confirmed 100% logical validity and hallucination-free generation, with only a minor 2% incompleteness in some answers.
Also Read:
- BinaryShield: Protecting AI Services with Privacy-Preserving Attack Fingerprints
- AI-Powered Scambaiting: A Deep Dive into Engaging and Disrupting Online Fraud
Key Strengths and Future Directions
The ForensicsData dataset and its generation methodology offer several significant strengths. It provides realistic behavioral coverage across numerous malware categories, is current with modern malware techniques, and delivers structured, validated outputs. The use of multiple LLMs enhances diversity, and the ethical generation of descriptive Q-C-A triplets (rather than executable code) maintains integrity. While some models showed instruction sensitivity and output quality depended on the original report content, proactive measures like prompt constraints and LLM-as-Judge filtering effectively mitigated hallucination risks.
ForensicsData represents a significant step forward in digital forensics research, providing a foundation for developing intelligent forensic tools and enabling reproducible experiments. Future work will focus on fine-tuning LLMs for specific forensic tasks, developing Retrieval-Augmented Generation systems, exploring multi-agent analysis, and expanding the dataset to cover more platforms and larger sample sizes. For more detailed information, you can refer to the original research paper.
