TLDR: Researchers at North Carolina State University have developed a new adversarial attack called RisingAttacK, capable of manipulating AI computer vision systems to misinterpret images with minimal, imperceptible changes. This technique poses significant risks for AI applications in autonomous vehicles, health technologies, and security, highlighting critical vulnerabilities in current AI security.
Raleigh, NC – July 1, 2025 – A groundbreaking and concerning new adversarial attack, dubbed "RisingAttacK," has been unveiled by researchers at North Carolina State University. This innovative technique allows malicious actors to subtly manipulate artificial intelligence (AI) computer vision systems, causing them to "see" whatever the attacker intends, even when the altered images appear identical to the human eye. The findings underscore significant vulnerabilities in widely used AI vision models and raise urgent questions about the security of AI applications in critical sectors.
The research, led by Tianfu Wu, an associate professor of electrical and computer engineering at NC State, and Thomas Paniagua, a recent Ph.D. graduate, demonstrates that RisingAttacK is effective against all of the most commonly deployed AI computer vision systems, including ResNet-50, DenseNet-121, ViTB, and DEiT-B.
"We wanted to find an effective way of hacking AI vision systems because these vision systems are often used in contexts that can affect human health and safety – from autonomous vehicles to health technologies to security applications," stated Professor Wu. He emphasized the critical importance of securing these AI systems and noted that "identifying vulnerabilities is an important step in making these systems secure, since you must identify a vulnerability in order to defend against it."
RisingAttacK operates through a series of sophisticated operations designed to make the fewest possible changes to an image while achieving the attacker’s goal. The process involves:
1. Identifying Visual Features: The technique first pinpoints all visual features within an image.
2. Determining Key Features: It then identifies which of these features are most crucial for the AI to achieve a specific recognition task. For instance, if the goal is to prevent an AI from identifying a car, RisingAttacK determines the features most vital for car recognition.
3. Calculating Sensitivity: Finally, the system calculates the AI’s sensitivity to data changes, particularly within these key features. "This requires some computational power, but allows us to make very small, targeted changes to the key features that makes the attack successful," Wu explained.
The practical implications are stark: "The end result is that two images may look identical to human eyes, and we might clearly see a car in both images. But due to RisingAttacK, the AI would see a car in the first image but would not see a car in the second image," Wu elaborated. This capability extends to influencing the AI’s ability to identify any of the top 20 or 30 targets it was trained on, such as pedestrians, bicycles, or stop signs.
The potential real-world consequences are far-reaching. Adversarial attacks like RisingAttacK could be exploited to manipulate autonomous vehicles into misidentifying traffic signals or pedestrians, or to compromise medical imaging systems, leading to inaccurate diagnoses.
The paper, titled "Adversarial Perturbations Are Formed by Iteratively Learning Linear Combinations of the Right Singular Vectors of the Adversarial Jacobian," was presented on July 15 at the International Conference of Machine Learning in Vancouver, Canada. Co-authored by Ph.D. student Chinmay Savadikar, the work received support from the National Science Foundation and the Army Research Office.
Also Read:
- ChatGPT Agent Successfully Bypasses Cloudflare CAPTCHA, Sparking Cybersecurity Alarm
- AI-Powered Disinformation Surges as U.S. Resources to Combat It Dwindle
The research team has proactively made RisingAttacK publicly available on GitHub, encouraging the broader research community to utilize it for testing neural network vulnerabilities. This move aims to foster the development of robust defense mechanisms against such sophisticated attacks. Professor Wu concluded, "Moving forward, the goal is to develop techniques that can successfully defend against such attacks."
