Assessing Large Language Models for Advanced Malware Auditing

TLDR: A new framework called MalEval evaluates how well Large Language Models (LLMs) can perform detailed Android malware behavior auditing, going beyond simple classification. It uses expert-verified data, reduces code noise, and defines four tasks (function prioritization, evidence attribution, behavior synthesis, and sample discrimination) to assess LLMs. While LLMs show promise in identifying malicious patterns and partially reducing analyst workload, they currently struggle with complex reasoning, coherent attack chain reconstruction, and reliable evidence attribution, indicating significant challenges remain for trustworthy fine-grained malware auditing.

Automated systems are great at flagging potential malware, but understanding *what* the malware does and *how* it does it remains a significant challenge for security analysts. This crucial step, known as malware behavior auditing, is often slow and costly due to the complex and hidden nature of malicious code. Large Language Models (LLMs) offer a promising new approach, but their effectiveness in this area has been largely unexamined.

A recent research paper, titled “Beyond Classification: Evaluating LLMs for Fine-Grained Automatic Malware Behavior Auditing,” introduces a comprehensive evaluation framework called MalEval. Developed by Xinran Zheng, Xingzhi Qian, Yiling He, Shuo Yang, and Lorenzo Cavallaro, this framework is designed to assess how well LLMs can support detailed malware auditing under real-world conditions. You can read the full paper here.

The Core Problem: Beyond Simple Detection

Traditional malware detection often stops at classification – simply identifying a file as malicious. However, auditing requires a deeper dive: providing verifiable explanations of malicious behavior, substantiating claims with evidence, and filtering out false alarms. Current LLM applications face three main hurdles in this domain: a lack of detailed behavioral and evidence annotations for proper assessment, interference from abundant benign code, and the risk of untraceable or fabricated outputs (hallucinations) that undermine credibility.

Introducing MalEval: A Framework for Deeper Understanding

MalEval tackles these limitations by breaking down the complex auditing process into a structured workflow. It focuses on three key areas:

• Verifiable Ground Truth Generation: To overcome data scarcity, MalEval curates a diverse dataset of Android applications, including both archived and the latest malware samples, as well as benign apps that might be misclassified. This dataset comes with manually verified low-level indicators (sensitive API lists) and high-level, expert-written behavior reports, ensuring a reliable foundation for evaluation.
• Code Context Reduction: To manage the vast amount of code and reduce noise, MalEval employs static reachability analysis. This process identifies and extracts only the potentially executable functions, significantly narrowing the scope of analysis to the most relevant code.
• Fine-grained Traceable and Verifiable Evaluation: Instead of feeding raw code, MalEval represents each relevant function as an “intermediate structural unit.” This unit includes the function’s signature, a context-driven summary of its role, and a sensitivity score. This approach provides consistent and traceable anchors for evaluation, allowing for a more precise assessment of LLM performance.

Four Key Auditing Tasks

MalEval defines four analyst-aligned evaluation tasks to mirror a human auditor’s workflow:

1. Function Prioritization: Evaluates an LLM’s ability to highlight high-risk functions hidden among thousands of benign ones.
2. Evidence Attribution: Measures if an LLM can correctly identify functions that invoke sensitive APIs, linking low-level evidence to potential malicious intent.
3. Behavioral Synthesis: Tests the LLM’s capacity to transform dispersed technical evidence into a concise, logically consistent, and human-readable narrative report.
4. Sample Discrimination: Examines whether LLMs can distinguish misclassified benign samples from genuine threats, acting as a corrective safeguard.

A unified metric, the Workload Reduction Score (WRS), quantifies the reduction in manual effort achieved by using LLMs in security operations.

Key Findings and Limitations

The evaluation of seven widely used LLMs (including Llama-3.1-8B-Instruct, Qwen2.5-Coder-14B-Instruct, Qwen3-32B, DeepSeek-V3-0324, GPT-4o-mini, Gemini-2.5-Flash, and Claude-3.7-Sonnet) revealed both promising capabilities and significant limitations:

• Even the strongest model, Claude-3.7-Sonnet, achieved only a 50.67% WRS, indicating that LLMs are still far from fully reliable for malware auditing.
• LLMs struggle to consistently identify the few functions most critical to malicious classification (low Fidelity Score).
• While some models (like the Qwen family) are good at broad evidence coverage, others (like Claude-3.7-Sonnet) excel at pinpointing decisive functions, highlighting different attribution styles.
• Gemini-2.5-Flash produced the most comprehensive and coherent reports, while Claude-3.7-Sonnet performed best in distinguishing true threats from false alarms. Smaller models like Llama-3.1-8B largely failed across most metrics.
• Inter-procedural context (understanding how functions interact) is crucial for identifying decisive functions and synthesizing coherent behaviors. Removing this context significantly degrades performance.
• Adding application-level metadata (like package or certificate information) had mixed effects. Some models (Deepseek-V3, Claude-3.7-Sonnet) improved, suggesting they can integrate this evidence, while others (Gemini-2.5-Flash) saw a drop in performance, indicating it might act as distracting noise.

Systematic Biases and Failure Modes

The research also uncovered systematic biases and common failure modes:

• LLMs consistently under-observed complex behaviors like “privilege escalation” and “stealthy escalation,” struggling with multi-step attack chains. They were better at explicit, API-driven behaviors like “SMS abuse” or “bank stealing.”
• Hallucination tendencies varied, with smaller models showing a higher propensity to generate novel or misaligned behaviors.
• Common failure cases included: inability to synthesize global threat objectives (missing the overall attack strategy), superficial technical interpretation (focusing on artifacts rather than causal exploit reasoning), benign-by-association bias (misattributing malicious actions to trusted apps), and logical blindness due to legal API abusing (failing to see malicious intent when legitimate APIs are combined for harmful purposes).

Also Read:

• Understanding Malware Behavior with Large Language Models: Introducing BEACON
• Unpacking LLMs in Penetration Testing: A Deep Dive into Performance and Potential

The Road Ahead

MalEval provides a reproducible framework and establishes a foundation for future research. While LLMs show potential in capturing typical malicious patterns and partially reducing analyst workload, their current capabilities are insufficient for trustworthy, fine-grained malware auditing. Significant advancements are needed in cross-function reasoning, behavior type calibration, and hallucination control to bridge the gap between statistical pattern recognition and the adversarial reasoning required for cybersecurity.