TLDR: The MeAJOR Corpus is a new, large-scale dataset of 135,894 phishing and legitimate emails designed to improve the accuracy of machine learning models in detecting phishing threats. It combines multiple open-source datasets and features extensive engineering and cleaning, addressing common issues like data imbalance and limited feature coverage in existing resources. Experiments show that models trained on MeAJOR, particularly XGBoost, achieve high F1 scores (up to 98.34%), demonstrating its effectiveness in building robust phishing detection systems.
Phishing emails continue to be a major threat in the digital world, tricking people with deceptive content and harmful links or attachments. While machine learning models are quite good at spotting these threats, their effectiveness heavily depends on the quality and variety of the data they are trained on.
Existing datasets for phishing detection often have significant limitations. These include an uneven distribution of phishing versus legitimate emails (class imbalance), a lack of diverse features extracted from emails, narrow coverage of different phishing tactics, outdated samples, and inconsistent ways of preparing the data. Many studies also rely on private or small datasets, which makes it hard to generalize their findings or reproduce their results.
Introducing the MeAJOR Corpus
To address these challenges, researchers have developed the MeAJOR (Merged email Assets from Joint Open-source Repositories) Corpus. This is a new, multi-source dataset specifically designed for detecting phishing emails. It combines five different open-source datasets, resulting in a comprehensive collection of 135,894 email samples. This large and diverse collection includes a wide range of phishing tactics and legitimate emails, making it a robust resource for training advanced detection models.
The creation of the MeAJOR Corpus involved a meticulous process of feature engineering and data cleaning. Features were extracted from six main parts of an email: the body text, embedded URLs, attachments, message headers, HTML structure, and external domain reputation. These features capture both technical anomalies (like mismatched sender information) and social engineering cues (like urgent language). For instance, URL features include details like length, subdomain count, and the presence of suspicious characters, which are crucial indicators of malicious links.
The data cleaning process was essential to ensure the quality and reliability of the dataset. It involved decoding various character encodings, stripping HTML markup to get clean text, removing quoted replies and irrelevant decorations, and normalizing characters. Crucially, sensitive information like email addresses and URLs were masked with placeholder tags to protect privacy while still preserving the structural signals important for detection.
Evaluating the Dataset’s Performance
To demonstrate the utility of the MeAJOR Corpus, the researchers conducted systematic experiments using four popular machine learning and deep learning models: Random Forest (RF), Extreme Gradient Boosting (XGBoost), Multilayer Perceptron (MLP), and Convolutional Neural Network (CNN). Each model was tested with different combinations of features: text only, text plus URL features, text plus attachment features, and a combination of all three.
The results were compelling. XGBoost consistently showed strong performance across all feature combinations, achieving an impressive F1-score of 98.34% when trained on the Text + URL Set. This performance is highly competitive with state-of-the-art results in the field, even with simpler feature engineering. The significant improvement observed when URL features were included highlights their critical importance in identifying phishing attempts. While attachment features had a more neutral impact, their inclusion supports the trend of using multiple types of information for detection.
The deep learning models, CNN and MLP, also performed well, especially with enriched feature sets. This confirms that deep learning is highly applicable for analyzing textual features in phishing detection. The strong performance of models trained solely on text also affirms the quality of the linguistic data preserved through the preprocessing steps.
Also Read:
- Large Language Models: A New Frontier in Information Security
- Enhancing Cyber Threat Intelligence Mapping with AI-Generated Data
A Foundation for Future Research
The MeAJOR Corpus provides a standardized, feature-rich, and scalable resource that addresses many limitations of previous datasets. By offering both raw data for custom feature extraction and ready-to-use engineered features, it provides great flexibility for researchers. Its design inherently supports a multi-modal approach, integrating signals from various technical, structural, and linguistic dimensions.
This novel dataset is expected to be a valuable foundation for future research and development in phishing detection. It can be used to explore advanced techniques, such as large language models, transformer architectures, and multi-modal data fusion, ultimately leading to more robust and accurate systems for safeguarding users against evolving email threats. You can find more details about this research in the full paper: MeAJOR Corpus: A Multi-Source Dataset for Phishing Email Detection.
