TLDR: A study evaluated five NIDS datasets (CIC-IDS2017, SWaT, WADI, Sherlock, CIC-Modbus2023) against 94 energy sector MITRE ATT&CK techniques, finding significant coverage gaps, especially in lateral movement and industrial protocol manipulation. While individual datasets like Sherlock and CIC-IDS2017 performed best, combining CIC-IDS2017, Sherlock, and CIC-Modbus2023 achieved 92% coverage, highlighting the need for enhanced, combined datasets to effectively protect converged IT/OT energy infrastructures.
Cyberattacks on critical infrastructure, especially in the energy sector, are becoming more frequent and sophisticated. These attacks can cause significant physical damage, as seen in incidents like Stuxnet, Triton, and the Sandworm attacks on the Ukrainian power grid. A major factor contributing to this vulnerability is the increasing convergence of Information Technology (IT) and Operational Technology (OT) within energy systems. This convergence, combined with legacy equipment and extended lifecycles, expands the potential targets for cyber threats.
Network Intrusion Detection Systems (NIDS) are crucial for defending these industrial environments. However, a recent study titled Mind the Gap: Missing Cyber Threat Coverage in NIDS Datasets for the Energy Sector highlights a significant problem: NIDS are often developed using publicly available datasets that primarily focus on enterprise IT environments. This raises serious concerns about their effectiveness in the unique, combined IT/OT settings of energy infrastructures.
The research, conducted by Adrita Rahman Tory, Khondokar Fida Hasan, Md Saifur Rahman, Nickolaos Koroniotis, and Mohammad Ali Moni, evaluated five widely used datasets: CIC-IDS2017, SWaT, WADI, Sherlock, and CIC-Modbus2023. The goal was to see how well these datasets represent real-world cyber threats in the energy sector. The study focused on 94 network-detectable attack techniques identified from 274 documented energy sector incidents, all mapped to the MITRE ATT&CK framework.
The researchers used a structured five-step analytical approach. This involved collecting threat intelligence, assessing and prioritizing risks, filtering for network-detectable techniques, building a knowledge base for each dataset, and finally, automatically mapping dataset coverage using a dual large language model (LLM) assessment to ensure reliability.
The findings revealed significant gaps. Individually, the Sherlock dataset showed the highest mean coverage at 0.56, closely followed by CIC-IDS2017 at 0.55. In contrast, SWaT and WADI recorded the lowest scores at 0.38. This indicates that while some datasets are better than others, none provide comprehensive coverage on their own.
A key insight from the study is the complementary nature of these datasets. When combined, CIC-IDS2017, Sherlock, and CIC-Modbus2023 achieved an impressive aggregate coverage of 92% of the identified threat techniques. Using all five datasets pushed the coverage to 94%, demonstrating that a combination of resources is necessary for a more complete picture of cyber threats.
However, the analysis also pinpointed critical gaps. Techniques related to lateral movement, such as “Remote Services: SMB/Windows Admin Shares,” and industrial protocol manipulation, like “Program PLC” or “Manipulate Control,” showed minimal to no coverage across the datasets. These are particularly concerning because they are frequently observed in actual energy sector attacks, such as the Ukrainian power grid incidents.
Also Read:
- Collaborative AI Strengthens Ransomware Defense While Protecting Data Privacy
- Securing In-Vehicle Networks with Efficient Binarized Neural Networks
The study concludes that there’s a pressing need to enhance existing datasets to better reflect the complex, hybrid IT/OT environments of the energy sector. This includes improving representation of industrial protocols and cross-domain attack paths. The methodology proposed in this paper offers a clear pathway for future dataset development, aiming for more robust NIDS evaluations and ultimately, stronger cybersecurity defenses for critical energy infrastructures.
