TLDR: A new paper demonstrates that audio-based large language models (ALLMs) like Qwen2-Audio are vulnerable to “adversarial noise.” Attackers can craft subtle audio perturbations to either force ALLMs to perform specific malicious actions (like deleting calendar events) or degrade their performance, even in real-world scenarios by playing sounds over the air. While some defenses exist, attackers can adapt, highlighting a significant security risk for AI voice assistants.
A recent research paper titled “Attacker’s Noise Can Manipulate Your Audio-based LLM in the Real World” sheds light on critical vulnerabilities in audio-based large language models (ALLMs), such as Qwen2-Audio. Authored by Vinu Sankar Sadasivan, Soheil Feizi, Rajiv Mathews, and Lun Wang, this study reveals how subtle, crafted audio disturbances can be used to manipulate these advanced AI systems, posing a significant threat in real-world scenarios.
The paper identifies two main types of attacks: targeted and untargeted. In a targeted attack, an adversary creates stealthy audio perturbations designed to force an ALLM to perform specific, predefined actions. For instance, an attacker could play background noise that tricks an AI assistant into waking up (e.g., by interpreting the noise as “Hey Qwen”) or even executing harmful commands like “delete my calendar events” or “send money to X.” The researchers demonstrated a 100% success rate in these targeted attacks across various experimental settings, highlighting the ease with which these models can be exploited.
Untargeted attacks, on the other hand, aim to degrade the ALLM’s overall utility. By injecting adversarial noise during a user’s interaction, the attacker can significantly impair the model’s ability to understand and respond correctly. This could lead to a voice assistant misinterpreting commands or generating nonsensical replies, making the service unreliable for innocent users. The study showed that adversarial noise caused much more extreme degradation in speech recognition performance compared to random noise.
A crucial aspect of this research is its focus on real-world applicability. The authors explain that these adversarial sounds can be transmitted over the air, impacting other users in public spaces without their knowledge. To achieve this, they incorporated audio augmentation techniques like translation, additive noise, and SpecAugment during the attack optimization process. SpecAugment, in particular, proved vital for making the attacks robust against real-world distortions like microphone interference and ambient noise, enabling successful manipulation even when played through a speaker and recorded by another device.
The research also explored the transferability of these attacks, finding that they remain effective even when the ALLM is given different system instructions, including those designed to make it ignore background noise. This suggests that custom instructions or basic filtering might not be sufficient to protect against these sophisticated attacks.
While the paper highlights a significant security concern, it also investigates potential defensive measures. Simple input audio augmentations, such as sample rate modification and noise reduction, were tested. However, the most effective defense identified was neural audio compression, specifically EnCodec. This technique showed nearly 100% success in countering the attacks. Despite this, the authors caution that attackers could adapt their methods to bypass these defenses if they are aware of them, emphasizing the need for continuous development of more robust and adaptive security protocols.
Also Read:
- The Hidden Threat: LLM Agents and Complete Computer Takeover
- Unmasking the Flaw in LLM Prompt Injection Detection: A New Attack Evades State-of-the-Art Defenses
In conclusion, this paper underscores a critical vulnerability in audio-based LLMs, demonstrating how adversarial audio can be used for both targeted malicious actions and general degradation of service in real-world scenarios. The findings call for increased caution in deploying and open-sourcing such models and stress the urgent need for advanced, adaptive defenses to protect users from these emerging threats. For more details, you can read the full research paper here.
